Failed vCloud 1.5.2 to 5.1.2 upgrade

While upgrading our final vCloud 1.5 install to 5.1.2, we ran into an issue while upgrading the vCloud database.  The upgrade log in $VCLOUD_HOME/logs/ contained the following

2013-09-20 10:21:45,658 | DEBUG    | pool-1-thread-1           | RawSQLTask                     | Executing sql ‘
DECLARE num_rows integer;
BEGIN
SELECT count(*)
INTO num_rows
FROM all_constraints
WHERE constraint_name = ‘FK_NETWORK_POOL’;

IF num_rows <> 0 THEN
EXECUTE IMMEDIATE ‘ALTER TABLE org_prov_vdc DROP CONSTRAINT fk_network_pool’;
END IF;
END;
‘ |
2013-09-20 10:21:46,163 | WARN     | pool-1-thread-1           | SerialAggregateTask            | Steps to upgrade to 2.0.238: Task failed due to uncaught exception |
java.sql.SQLException: ORA-02443: Cannot drop constraint  – nonexistent constraint
ORA-06512: at line 10

VMware supported provided the following database edit that resolved the issue.  Remember to commit the changes if you’re using Oracle.Confirm the database.schema.version value by running this query.

Confirm the database.schema.version value by running this query:

SELECT value FROM config WHERE name=’database.schema.version’;

This query should return 2.0.237.transition.

Run this script:

DECLARE
 no_constraint EXCEPTION;
 PRAGMA EXCEPTION_INIT(no_constraint, -2443);
BEGIN
 EXECUTE IMMEDIATE ‘ALTER TABLE network_assigned_mac DROP CONSTRAINT fk_namac_to_resource DROP INDEX’;
 EXCEPTION WHEN no_constraint THEN
   NULL;
END;

DECLARE
 no_constraint EXCEPTION;
 PRAGMA EXCEPTION_INIT(no_constraint, -2443);
BEGIN
 EXECUTE IMMEDIATE ‘ALTER TABLE network_assigned_mac DROP CONSTRAINT fk_namac_to_ri DROP INDEX’;
 EXCEPTION WHEN no_constraint THEN
   NULL;
END;

UPDATE config SET value=’2.0.238′ WHERE name=’database.schema.version’;

Re-run the ./bin/upgrade utility to complete the upgrade process.

Advertisements

Unable to add user to local admin group during guest customization

As part as our provisioning process on Windows VMs in vCloud, we modify the guest customization section of VMs to add the provisioning user to the local administrators group.  A stripped down version of the customization section on the VM looks like:
if “%1%” == “precustomization” (
goto end
) else if “%1%” == “postcustomization” (
  net localgroup administrators domain\user /add
)
:end
When the VM finished customizing, the provisioning user was not added to the local administrators group and the guest customization log (C:\windows\temp\customization-guest.log) file would contain the error:
System Error 1789 has occurred. The trust relationship between this workstation and the primary domain failed.
I could then manually add the user to the local administrators group with the command “net localgroup administrators domain\user /add” and received no error.

The VM was being joined to the domain via vCloud’s Domain Join functionality.

Since I was able to add the user manually, the trust relationship appeared to be OK and it seemed to be a timing issue.   To test this I put a delay before attempting to add the user to the group:
if “%1%” == “precustomization” (
goto end
) else if “%1%” == “postcustomization” (
  timeout 15
net localgroup administrators domain\user /add
)
:end
The user was then successfully added to the group and the error hasn’t came back since.

All NICs are not ready yet. Will try again after 10000 milliseconds

I recently ran into an issue with vCloud and Windows 2008 R2 guest customization where the guest customization log (C:\Windows\temp\customize-guest.log) would be flooded with the error “All NICs are not ready yet.  Will try again after 10000 milliseconds” and the VM would not be customized.

guest-customization

Here you can see that VMware tools was giving the VM an IP of 192.168.187.236.  This is odd because this isn’t the IP that vCloud had for the VM, and the IP comes from another environment.  I was told that the VM was moved from one environment to another but did not receive any more detail.

If you inspected the VM’s OVF environment with vmtoolsd, it would report the invalid IP.  The command to do this is “C:\Program  Files\VMware\VMware Tools\vmtoolsd.exe” –cmd “info-get guestinfo.ovfEnv”.  I don’t have a screenshot, but when trying to instantiate a new vApp from this corrupted template, the network selection of the wizard had an odd layout that I’ve never seen before.

What was happening was that the OVF environment on the VM was stuck with the old OVF info and vCloud wasn’t replacing it with the new info.   VMware tools was then configuring the VM with an IP that wasn’t valid for the network that the VM was on and the OS would report that the NIC wasn’t ready.  Eventually the guest customization process would time out and fail.

In an attempt to clear the OVF environment I performed and OVF export/import of the VM and re-imported it into vCloud, but this did not help.  I eventually resolved the issue by creating a new shell VM with no disk, attached the problem VM’s disk to it and imported the VM back into vCloud.

If anyone knows how to inspect a VM’s OVF environment from outside of the guest and/or how to clear it out, I’d be interested in hearing about it.