Posts Tagged ‘chef’

http://docs.chef.io/server/install_server.html#standalone does not mention how to configure knife.rb which is described on https://docs.chef.io/config_rb_knife.html . There is a outstanding difference of knife.rb for Chef Server 11 and 12:

In Chef Server 11:  chef_server_url 'http://localhost:4000/
In Chef Server 12:  chef_server_url 'http://localhost:4000/organizations/ORG_NAME

And the admin user in Chef Server 12 is named pivotal.  You only need to create a new user and a new ORG owned by this new user.

Here is a full install guide https://www.digitalocean.com/community/tutorials/how-to-set-up-a-chef-12-configuration-management-system-on-ubuntu-14-04-servers

Known issue : https://github.com/chef/chef-server/issues/50 Impossible to upload cookbooks if non-standard port is used


Read Full Post »

The VMware vSphere Big Data Extension product (aka Project Serengeti) needs to ensure the bootstrap order when deploying a hadoop cluster in which the slave nodes must wait for the master node.  BDE uses the native Chef way to implement it.

For example, hadoop datanode service should be started only after hadoop namenode service is started. In the chef recipe, we use this code to do this wait:

in namenode recipe github.com/vmwa…amenode.rb#L127 :

service ‘hadoop-namenode’ do

 action [:start]




in datanode recipe github.com/vmwa…datanode.rb#L29 :


service ‘hadoop-datanode’ do

 action [:start]


provide_service and wait_for_service are functions defined in the chef cookbook cluster_server_discovery.

The elastic_search and etcd or other service discovery solution can also be used in Chef cookbook.

Read Full Post »

As a code committer for Open Source Project Chef, I attended the 2nd Opscode Community Summit 2012 on October 23 & 24 in Seattle, WA.  Here is my notes for this summit.


  • about 100 persons attended this summit
  • a few SMB company and startups attended this summit, learning Chef and sharing achievements/techniques/questions with the community, including Opscode, CloudScaling, DreamHost, Blue Box, HP, Google,  Adobe, ThoughtWorks, Workday, LivingSocial, Qualcomm, Mopub, Nordstrom, OneHealth, Wikia, Fidelity, Heavy Water, thePlatform, etc.


  • about 46 one-hour sessions are hold (see session ; about 10~15 attendees for each session; Opscode holds about 1/3 sessions while Dreamhost, CloudScaling and others hold other sessions.
  • I hold one session ‘How VMware Serengeti uses Chef to Provision a Hadoop Cluster in a vSphere Cloud’, for introducing Serengeti, Ironfan and Chef techniques used in Serengeti.  Luckily about 6 persons are interested in my session.  They’re interested in:
    • architecture of Serengeti Ironfan?
    • what’s special in Hadoop and vSphere?
    • can Hadoop ran well in vSphere as in physical hosts?
    • any benchmarking?
  • Some hot topics
    • Chef Orchestration Layer (such as Ironfan)
    • Bare Metal Provisioning Tool (such as Razor from EMC + PuppetLabs, FAI)
    • Windows support of Chef
    • Chef Driven Monitoring and Metrics (e.g. Nagios)
    • TDD for writing Chef Cookbooks
    • Cookbook versioning
    • Chef Knife plugins for OpenStack and EC2
    • etc.


Serengeti and HVE

  • A Opscode Sales Manager is very interested in Serengeti and HVE, and told me that almost all their customers who runs hadoop are running it on physical machines rather than virtual machines.
  • She cares about the performance comparison of Hadoop+HVE on vSphere and Hadoop on physical hosts.
  • A CTO of a startup in Silicon Valley shows strong interests of Serengeti/Hadoop and Ironfan.


  • Ironfan/Infochimps has good reputation in Chef world (although there are a few people are not familiar with it)
  • Ironfan does a good job as Chef Orchestration tool. In the session ‘Chef Orchestration Layer’, users are eager to find an powerful Chef Orchestration layer, and find a solution to manager Chef nodes, control the bootstrapping workflow, handle cluster service dependency to resolve racing condition.   I introduced the appoach used in Serengeti and are well received.
  • seems there is no other orchestration tools (I only heard 2 tool in this summit) better than Ironfan in Chef world.


  • EMC and PuppetLab’s Project Razor (released in May 2012) can do physical and virtual hardware provisioning and software provisioning. Razor is open sourced under the Apache 2.0 and is fully compatible with both Puppet open source and Puppet Enterprise.
  • Some SMB and startups are using Physical Hosts + Chef to provision their infrastructure (such as Wikia)

Data Director

  • Fidelity uses a software for provisioning mysql and postgresql db (but not Oracle), sounds like very similar to Data Director. Maybe Fidelity company can become a potential customer of DD.


  • Talked with Chef creator and Opscode co-founder Adam Jacob about adding a  rollback feature (CHEF-3502) into Chef.
  • Talked with the organizer of Opscode Community Summit 2012 (also the Opscode co-founder) Jessie Robbins about Serengeti.
  • Talked with Joshua Timberman (Opscode Technical Community Manager) about adding a new feature (CHEF-3571) into Chef, and we are collaborating on driving the feature into next version of Chef.
  • etc.

Thanks very much!

Read Full Post »

Solution: compact couchdb file (/var/lib/couchdb/chef.couch) used by Chef Server

$ curl -H “Content-Type: application/json” -X POST http://localhost:5984/chef/_compact

ref: http://wiki.opscode.com/display/chef/CouchDB+Administration+for+Chef+Server

Read Full Post »