I was at VMWorld last week. Unfortunately, my network connectivity was limited so I was unable to post my notes during the conference.

Keynotes – John Chambers, CEO Cisco

John Chambers message was basically talking about how IT enables business and how to present that case to business executives.

Keynotes– Energy Panel

Quentin Hardy, Panel Moderator, Forbes Magazine
David Brooks, Harvard
Ray Cline, EDS
Amdrew Fanara, EPA
John Gibson, HSBC Bank

ESX Server CPU Scheduling

Andrei Drofeev, VMWare

CPU Scheduling Internals

  • SMP Virtual Machines
  • Load Balancing & Migrations
  • Hyperthreading Support
  • User Worlds – VMWare term for whole VM
  • Interrupts
  • NUMA

Performance Tips

  • Idle VMs still receive timer interrupts
  • Avoid VM CPU affinity
  • SMP – use as few virtual CPUs as possible
  • 64 bit guest generally better performance
  • Avoid running programs in service console
  • Don't fully commit CPU with reservations

ESXTOP Utility

  • High %RDY time – VM was ready for CPU time but had to wait
  • High %Wait / %Idle – workload is not CPU intensive

Future Directions

  • Scalability – # CPUs per host
  • Power mgt
  • Better multi-core support

VM Infrastructure 3 – Best Practices for Performance

Jeff Buell, Staff Engineer
Deraki Kulkarni, Sr. MTS, VMWare

Sources of Virtualization Overhead

  • CPU
  • Memory
  • Devices
  • Resource Management

CPU Performance:

  • Keep in mind OS timer interrupts. For awhile, Linx was using a 1000 Hz which was too often. Latest linux is set to 250 Hz which is OK. Windows is 100 Hz.
  • 64 bit guests give better performance
  • DIsable unused controllers / devices (see KB 1290)

Memory Performance

  • Page sharing – pages which are the same between VMs can be shared between VMs
  • Memory ballooning – require VMWare tools
  • Avoid active memory over commit (eliminate ESX memory swapping)
  • Right size guest OS memory


  • Ensure there is enough CPU to process networks
  • Use vmxnet network driver from VMWare tools

Storage Performance

  • Guest driver setting can affect performance (KB 9645697) – Increase the size of writes
  • Recommend using Fibre channel SAN
  • NFS or iSCSI – use more CPY than SAN
  • Better performance if use Virtual Center to create partitions
  • Increase VMs max outstanding disk requests if needed (KB 1268)

Using the Secure Technology Implementation Guide (STIG) with VMWare Infrastructure 3

DISA – Defense Infromation Systems Agency

STIG – general guide for securing systems

Vulnerability Categories
Category I – Worse level. Allows access tot he machine
Category II – Provides information that could lead to access (high potential)
Category III – Provide information that could lead to access

5 STIGs related to VMI3 (http://iase.disa.mil/stigs/index.html)

  • Virtual Computing STIG
  • Unix STIG (ESX Server, VMs)
  • DB STIG (Virtual Center)
  • Win OS STIG (Win VM)
  • Web Server STIG and checklist
  • Other => SRR scripts, associated checklists, vulnerabilities management systems

Run SRR scripts (system readiness review – ./start-SRR)

See slides for many details of current findings and their meanings.

VMWare Infrastructure 3: Advanced Diagnostic Log Analysis

Mostafa Khalil, VCP, VMWare Product Support Engineer

ESX Server Boot Process: Boot Loader -> initrd -> vmkernel -> vmnix -> /sbin/init ->init scripts -> vmware init scripts

Collecting logs:

  • UI via VI Client (right click on item and select export diagnostics data)
  • Multiple logs on each ESX server

VMKernel Log:

All events generated by vmkernel (warnings are also written to seperate log)
Logs rotates. All events since last load are also in memory at /proc/vmware/log.

General Log msg: timestamp, hostname, msg source, uptime, <instance>, device, src line #, msg

Translating vmkernel error codes: already listed in msg in ESX 3

Message Log:

Like linux messages
Console events, logon events, iSCSI authentication events


vi client events
Events done on behalf of various services


Events of interactions with Virtual Center


rule events

oldconf files

backup of config files modified by VC, VI Client or esxcfg-* scripts


Updates from esxupdates


Whenever kernel is loaded / updated

Security Architecture Design and Hardening of VMWare Infrastructure 3

Kick Larsen – Engineering – Product Security Officer
Banjot S. Chanana – Product Mgr, Platform Security
Brian Cosker-Swerske – Senior Consultant

Service Console

Actually a VM based on RedHat Linux (own Virtual CPU)
Ports 902, 80, 443 and 22 open by default

Disable ctrl-alt-del
Require password for single user mode

service console network parameters
login banners

Secure Networks
Password policy for local user accounts
Password complexity

Limit root access using securetty
enable syslog
Change snmp community string from the default

Securing VMs

Same as a physical box
Remove unnecessary functions / services
Disable cut and paste