Passive building design takes advantage of local micro-climate characteristics in site planning and analysis of house construction. In a ten-year project, we built the Orchid House with the concept of a green core to effectively reduce the energy consumption of the house. We have equipped Orchid House with smart control for passive design using an Internet of Things (IoT) solution called HouseTalk. With HouseTalk, the passive mechanisms of Orchid House are intelligently controlled to offer indoor comforts. HouseTalk integrates all sensors and actuators of the Orchid House through a Graphical User Interface (GUI). This GUI allows the users to easily and directly access all IoT devices installed in the house through graphical icon representations, and conveniently implement smart algorithms for smart home. HouseTalk dynamically adjusts the formula for HVAC operation in the closed private living area, which is appropriate for thermal control. In the open public shared space (the green core of the Orchid House), HVAC will consume much more energy, and the passive design with non-thermodynamic cycle equipment such as mechanical ventilation, evaporative cooling system, or other non-thermodynamic cycle systems are more appropriate for thermal control. HouseTalk enhances passive design by using non-thermodynamic cycle system with low energy-consumption equipment to reduce the CO2 concentration, purifying the air to maintain oxygen concentrations ranging between 18% and 21%. Also, in the green core, HouseTalk can effectively improve the cooling effect by up to 77% without HVAC.The reported thermal sensation indicates that over 90% of tenants are satisfied by HouseTalk's HVAC control. We demonstrate the potential of enhancing air quality through the photosynthesis by indoor plants. HouseTalk can effectively speed up the reduction of carbon dioxide concentrations by 53%. The oxygen concentration for the HouseTalk scenario ranges between 18% and 21%, which is always higher than that of the baseline scenario.