Our ZCM sysadmin here finished an upgrade from 11.2.3a to 11.3a and then updated through to FTF Roll Up 1. It's running on Windows 2008 R2 with an Oracle 11.2.0.4.0 database back end.

While poking around in ZCC post-upgrade, I noticed the "Inventory" tab for all devices in the zone show up blank. Reports with inventory related data run fine and agents seem to have no problem scanning and uploading inventory scan results. I spoke with the ZCM sysadmin and we started digging on the server.


We found this in the zcc.log on the server which seems to occur the second we click the Inventory tab on a device:

[DEBUG] [06/04/2014 21:01:56.989] [1916] [ZENServer] [95] [CITS-ZCM$] [ZCC] [] [com.novell.zenworks.datamodel.exceptions.InternalD ataModelException: org.hibernate.exception.SQLGrammarException: ORA-00904: "MACHINE0_"."ITERATIONNO": invalid identifier

at com.novell.zenworks.datamodel.database.AbstractDat abaseTypeHelper.newStandardDataModelException(Abst ractDatabaseTypeHelper.java:456)
at com.novell.zenworks.datamodel.database.AbstractDat abaseTypeHelper.throwStandardDataModelException(Ab stractDatabaseTypeHelper.java:441)
at com.novell.zenworks.datamodel.session.hibernate.Ab stractHibernateSessionImpl.handleException(Abstrac tHibernateSessionImpl.java:244)
at com.novell.zenworks.datamodel.services.inventory.I nventoryImpl.getInventoryObject(InventoryImpl.java :3683)
at com.novell.zenworks.datamodel.services.inventory.I nventoryImpl.getInvMachineWithZENworksUID(Inventor yImpl.java:1731)
at com.novell.zenworks.admin.pages.devices.DeviceInve ntorySummary.populateSelectedSection(DeviceInvento rySummary.java:290)
at com.novell.zenworks.admin.pages.devices.DeviceInve ntorySummary.createChildControls(DeviceInventorySu mmary.java:107)
at com.novell.web.controls.WebControl.ensureChildCont rols(WebControl.java:1738)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2633)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2645)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2645)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2645)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2645)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2645)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2645)
at com.novell.web.controls.Page.internalProcessReques t(Page.java:749)
at com.novell.web.controls.Page.processRequest(Page.j ava:678)
at com.novell.zenworks.fw.web.internal.layout.Index.p rocessRequest(Index.java:689)
at com.novell.web.internal.controls.PageTag.doEndTag( PageTag.java:86)
at org.apache.jsp.jsp.index_jsp._jspService(index_jsp .java:324)
at org.apache.jasper.runtime.HttpJspBase.service(Http JspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:728)
at org.apache.jasper.servlet.JspServletWrapper.servic e(JspServletWrapper.java:432)
at org.apache.jasper.servlet.JspServlet.serviceJspFil e(JspServlet.java:390)
at org.apache.jasper.servlet.JspServlet.service(JspSe rvlet.java:334)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:728)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.inv oke(ApplicationDispatcher.java:749)
at org.apache.catalina.core.ApplicationDispatcher.pro cessRequest(ApplicationDispatcher.java:487)
at org.apache.catalina.core.ApplicationDispatcher.doF orward(ApplicationDispatcher.java:379)
at org.apache.catalina.core.ApplicationDispatcher.for ward(ApplicationDispatcher.java:339)
at com.novell.web.NJWCServlet.serviceImpl(NJWCServlet .java:97)
at com.novell.zenworks.fw.web.internal.ZENworksServle t.service(ZENworksServlet.java:87)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:728)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:210)
at com.patchlink.sapphire.web.pages.vulnerability.ses sion.HibernateSessionFilter.doFilter(HibernateSess ionFilter.java:75)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBas e.invoke(AuthenticatorBase.java:581)
at org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:171)
at com.googlecode.psiprobe.Tomcat70AgentValve.invoke( Tomcat70AgentValve.java:39)
at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:99)
at org.apache.catalina.core.StandardEngineValve.invok e(StandardEngineValve.java:118)
at com.novell.zenworks.tomcat.ZENRequestValve.invoke( ZENRequestValve.java:1331)
at org.apache.catalina.connector.CoyoteAdapter.servic e(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.p rocess(AbstractHttp11Processor.java:1009)
at org.apache.coyote.AbstractProtocol$AbstractConnect ionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProce ssor.run(JIoEndpoint.java:312)
at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.hibernate.exception.SQLGrammarException: ORA-00904: "MACHINE0_"."ITERATIONNO": invalid identifier

at org.hibernate.exception.internal.SQLExceptionTypeD elegate.convert(SQLExceptionTypeDelegate.java:82)
at org.hibernate.exception.internal.StandardSQLExcept ionConverter.convert(StandardSQLExceptionConverter .java:49)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.c onvert(SqlExceptionHelper.java:125)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.c onvert(SqlExceptionHelper.java:110)
at org.hibernate.engine.jdbc.internal.proxy.AbstractS tatementProxyHandler.continueInvocation(AbstractSt atementProxyHandler.java:129)
at org.hibernate.engine.jdbc.internal.proxy.AbstractP roxyHandler.invoke(AbstractProxyHandler.java:81)
at com.sun.proxy.$Proxy50.executeQuery(Unknown Source)
at org.hibernate.loader.Loader.getResultSet(Loader.ja va:1953)
at org.hibernate.loader.Loader.doQuery(Loader.java:82 9)
at org.hibernate.loader.Loader.doQueryAndInitializeNo nLazyCollections(Loader.java:289)
at org.hibernate.loader.Loader.doList(Loader.java:243 8)
at org.hibernate.loader.Loader.doList(Loader.java:242 4)
at org.hibernate.loader.Loader.listIgnoreQueryCache(L oader.java:2254)
at org.hibernate.loader.Loader.list(Loader.java:2249)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoa der.java:470)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl .list(QueryTranslatorImpl.java:355)
at org.hibernate.engine.query.spi.HQLQueryPlan.perfor mList(HQLQueryPlan.java:195)
at org.hibernate.internal.SessionImpl.list(SessionImp l.java:1248)
at org.hibernate.internal.QueryImpl.list(QueryImpl.ja va:101)
at org.hibernate.internal.AbstractQueryImpl.uniqueRes ult(AbstractQueryImpl.java:890)
at com.novell.zenworks.datamodel.services.inventory.I nventoryImpl.getInventoryObject(InventoryImpl.java :3678)
... 51 more
Caused by: java.sql.SQLSyntaxErrorException: ORA-00904: "MACHINE0_"."ITERATIONNO": invalid identifier

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoe r.java:440)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoe r.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall. java:837)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.jav a:445)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java: 191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:5 23)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4 CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeFor Describe(T4CPreparedStatement.java:863)
at oracle.jdbc.driver.OracleStatement.executeMaybeDes cribe(OracleStatement.java:1153)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTi meout(OracleStatement.java:1275)
at oracle.jdbc.driver.OraclePreparedStatement.execute Internal(OraclePreparedStatement.java:3576)
at oracle.jdbc.driver.OraclePreparedStatement.execute Query(OraclePreparedStatement.java:3620)
at oracle.jdbc.driver.OraclePreparedStatementWrapper. executeQuery(OraclePreparedStatementWrapper.java:1 491)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement .executeQuery(NewProxyPreparedStatement.java:454)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unkno wn Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.hibernate.engine.jdbc.internal.proxy.AbstractS tatementProxyHandler.continueInvocation(AbstractSt atementProxyHandler.java:122)
... 67 more
] [org.hibernate.exception.SQLGrammarException: ORA-00904: "MACHINE0_"."ITERATIONNO": invalid identifier
] [] [com.novell.zenworks.datamodel.exceptions.InternalD ataModelException] [ZENServer]
[DEBUG] [06/04/2014 21:01:56.989] [1916] [ZENServer] [95] [CITS-ZCM$] [ZCC] [] [com.novell.zenworks.datamodel.exceptions.InternalD ataModelException: org.hibernate.exception.SQLGrammarException: ORA-00904: "MACHINE0_"."ITERATIONNO": invalid identifier

at com.novell.zenworks.datamodel.database.AbstractDat abaseTypeHelper.newStandardDataModelException(Abst ractDatabaseTypeHelper.java:456)
at com.novell.zenworks.datamodel.database.AbstractDat abaseTypeHelper.throwStandardDataModelException(Ab stractDatabaseTypeHelper.java:441)
at com.novell.zenworks.datamodel.session.hibernate.Ab stractHibernateSessionImpl.handleException(Abstrac tHibernateSessionImpl.java:244)
at com.novell.zenworks.datamodel.services.inventory.I nventoryImpl.getInventoryObject(InventoryImpl.java :3683)
at com.novell.zenworks.datamodel.services.inventory.I nventoryImpl.getInvMachineWithZENworksUID(Inventor yImpl.java:1731)
at com.novell.zenworks.admin.pages.devices.DeviceInve ntorySummary.populateSelectedSection(DeviceInvento rySummary.java:290)
at com.novell.zenworks.admin.pages.devices.DeviceInve ntorySummary.prerender(DeviceInventorySummary.java :117)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2634)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2645)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2645)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2645)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2645)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2645)
at com.novell.web.controls.WebControl.preRenderRecurs iveInternal(WebControl.java:2645)
at com.novell.web.controls.Page.internalProcessReques t(Page.java:749)
at com.novell.web.controls.Page.processRequest(Page.j ava:678)
at com.novell.zenworks.fw.web.internal.layout.Index.p rocessRequest(Index.java:689)
at com.novell.web.internal.controls.PageTag.doEndTag( PageTag.java:86)
at org.apache.jsp.jsp.index_jsp._jspService(index_jsp .java:324)
at org.apache.jasper.runtime.HttpJspBase.service(Http JspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:728)
at org.apache.jasper.servlet.JspServletWrapper.servic e(JspServletWrapper.java:432)
at org.apache.jasper.servlet.JspServlet.serviceJspFil e(JspServlet.java:390)
at org.apache.jasper.servlet.JspServlet.service(JspSe rvlet.java:334)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:728)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.ApplicationDispatcher.inv oke(ApplicationDispatcher.java:749)
at org.apache.catalina.core.ApplicationDispatcher.pro cessRequest(ApplicationDispatcher.java:487)
at org.apache.catalina.core.ApplicationDispatcher.doF orward(ApplicationDispatcher.java:379)
at org.apache.catalina.core.ApplicationDispatcher.for ward(ApplicationDispatcher.java:339)
at com.novell.web.NJWCServlet.serviceImpl(NJWCServlet .java:97)
at com.novell.zenworks.fw.web.internal.ZENworksServle t.service(ZENworksServlet.java:87)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:728)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:210)
at com.patchlink.sapphire.web.pages.vulnerability.ses sion.HibernateSessionFilter.doFilter(HibernateSess ionFilter.java:75)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBas e.invoke(AuthenticatorBase.java:581)
at org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:171)
at com.googlecode.psiprobe.Tomcat70AgentValve.invoke( Tomcat70AgentValve.java:39)
at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:99)
at org.apache.catalina.core.StandardEngineValve.invok e(StandardEngineValve.java:118)
at com.novell.zenworks.tomcat.ZENRequestValve.invoke( ZENRequestValve.java:1331)
at org.apache.catalina.connector.CoyoteAdapter.servic e(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.p rocess(AbstractHttp11Processor.java:1009)
at org.apache.coyote.AbstractProtocol$AbstractConnect ionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProce ssor.run(JIoEndpoint.java:312)
at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: org.hibernate.exception.SQLGrammarException: ORA-00904: "MACHINE0_"."ITERATIONNO": invalid identifier

at org.hibernate.exception.internal.SQLExceptionTypeD elegate.convert(SQLExceptionTypeDelegate.java:82)
at org.hibernate.exception.internal.StandardSQLExcept ionConverter.convert(StandardSQLExceptionConverter .java:49)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.c onvert(SqlExceptionHelper.java:125)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.c onvert(SqlExceptionHelper.java:110)
at org.hibernate.engine.jdbc.internal.proxy.AbstractS tatementProxyHandler.continueInvocation(AbstractSt atementProxyHandler.java:129)
at org.hibernate.engine.jdbc.internal.proxy.AbstractP roxyHandler.invoke(AbstractProxyHandler.java:81)
at com.sun.proxy.$Proxy50.executeQuery(Unknown Source)
at org.hibernate.loader.Loader.getResultSet(Loader.ja va:1953)
at org.hibernate.loader.Loader.doQuery(Loader.java:82 9)
at org.hibernate.loader.Loader.doQueryAndInitializeNo nLazyCollections(Loader.java:289)
at org.hibernate.loader.Loader.doList(Loader.java:243 8)
at org.hibernate.loader.Loader.doList(Loader.java:242 4)
at org.hibernate.loader.Loader.listIgnoreQueryCache(L oader.java:2254)
at org.hibernate.loader.Loader.list(Loader.java:2249)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoa der.java:470)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl .list(QueryTranslatorImpl.java:355)
at org.hibernate.engine.query.spi.HQLQueryPlan.perfor mList(HQLQueryPlan.java:195)
at org.hibernate.internal.SessionImpl.list(SessionImp l.java:1248)
at org.hibernate.internal.QueryImpl.list(QueryImpl.ja va:101)
at org.hibernate.internal.AbstractQueryImpl.uniqueRes ult(AbstractQueryImpl.java:890)
at com.novell.zenworks.datamodel.services.inventory.I nventoryImpl.getInventoryObject(InventoryImpl.java :3678)
... 50 more
Caused by: java.sql.SQLSyntaxErrorException: ORA-00904: "MACHINE0_"."ITERATIONNO": invalid identifier

at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoe r.java:440)
at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoe r.java:396)
at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall. java:837)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.jav a:445)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java: 191)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:5 23)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4 CPreparedStatement.java:207)
at oracle.jdbc.driver.T4CPreparedStatement.executeFor Describe(T4CPreparedStatement.java:863)
at oracle.jdbc.driver.OracleStatement.executeMaybeDes cribe(OracleStatement.java:1153)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTi meout(OracleStatement.java:1275)
at oracle.jdbc.driver.OraclePreparedStatement.execute Internal(OraclePreparedStatement.java:3576)
at oracle.jdbc.driver.OraclePreparedStatement.execute Query(OraclePreparedStatement.java:3620)
at oracle.jdbc.driver.OraclePreparedStatementWrapper. executeQuery(OraclePreparedStatementWrapper.java:1 491)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement .executeQuery(NewProxyPreparedStatement.java:454)
at sun.reflect.GeneratedMethodAccessor30.invoke(Unkno wn Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.hibernate.engine.jdbc.internal.proxy.AbstractS tatementProxyHandler.continueInvocation(AbstractSt atementProxyHandler.java:122)
... 66 more
] [org.hibernate.exception.SQLGrammarException: ORA-00904: "MACHINE0_"."ITERATIONNO": invalid identifier
] [] [com.novell.zenworks.datamodel.exceptions.InternalD ataModelException] [ZENServer]
[DEBUG] [06/04/2014 21:01:57.5] [1916] [ZENServer] [95] [CITS-ZCM$] [ZCC] [] [Form.java CSRF TOKEN:53ff44c3e8f2a5a3e8e285bf540c0132 for the PageId:workstationInventory] [] [] [] [ZENServer]


Exact same thing shows up in the development environment as well, but it wasn't caught before we upgraded production (that one's on me). Doesn't appear to impede communications with agents or the like and inventory data appears to get to the server and can be seen in the first Reports page of a device report, but if you click on a computer name in the device report to get more detail the page goes blank with just the line: "ORA-00942: table or view does not exist"

Reviewing the server upgrade log, the only error logged was when it couldn't create a table in the new audit database. The upgrade completed, but the upgrade log file viewed to show "All messages" has a DEBUG entry of $INSTALL_SUCCESS$:NONFATAL_ERROR followed by $UPGRADE_RESULTS$:FAILED shortly after the table creation error.

Debugging the command found an issue where a seemingly validly formatted SQL command wouldn't execute, claiming an index tablespace wasn't specified when it actually appears to be in the command (relating to the PRIMARY KEY, tablespace for indexes is called INDEX). The DBA manually created the table except the primary key, then manually added the primary key which seemed to work fine unlike when it was automated. This happened in development as well, apparently. In both environments, audit functions appear to work fine and so does the Dashboard feature which feeds from the audit database. Here's the query that bombed:

CREATE TABLE DA_USERLOGIN (
EVENTID RAW(16) NOT NULL ENABLE,
CREATEDDATE TIMESTAMP (6) NOT NULL ENABLE,
EVENTMESSAGE VARCHAR2(255 CHAR),
EXPIRYDATE TIMESTAMP (6) NOT NULL ENABLE,
NOTIFICATIONREASON VARCHAR2(255 CHAR),
NOTIFICATIONSTATUS CHAR(1 CHAR) NOT NULL ENABLE,
WEIGHT NUMBER(10,0) NOT NULL ENABLE,
EVENTSEQID VARCHAR2(255 CHAR) NOT NULL ENABLE,
DEVICEID RAW(16) NOT NULL ENABLE,
EVENTTYPEID RAW(16) NOT NULL ENABLE,
INITIATOR RAW(16) NOT NULL ENABLE,
TARGETGUID RAW(16) NOT NULL ENABLE,
INITIATOR_TYPE varchar2(2 byte) default '00' NOT NULL ENABLE,
TARGETGUID_TYPE VARCHAR2(2 BYTE) DEFAULT '00' NOT NULL ENABLE,
PRIMARY KEY (EVENTID) USING INDEX TABLESPACE INDEX ENABLE,
CONSTRAINT FK_DA_USERLOGIN1 FOREIGN KEY (EVENTTYPEID) REFERENCES AUDIT_EVENT_DEF (EVENTTYPEID) ON DELETE CASCADE ENABLE,
CONSTRAINT FK_DA_USERLOGIN7 FOREIGN KEY (INITIATOR) REFERENCES USERLOGIN_REF (USERLOGINID) ON DELETE CASCADE ENABLE,
CONSTRAINT FK_DA_USERLOGIN6 FOREIGN KEY (TARGETGUID) REFERENCES ZDEVICE_REF (DEVICEID) ON DELETE CASCADE ENABLE,
CONSTRAINT FK_DA_USERLOGIN5 FOREIGN KEY (DEVICEID) REFERENCES ZDEVICE_REF (DEVICEID) ON DELETE CASCADE ENABLE )
TABLESPACE ZENAUDIT
PARTITION BY RANGE (CREATEDDATE)
SUBPARTITION BY HASH(EVENTTYPEID)
SUBPARTITIONS 16 (
PARTITION DUL201424 VALUES LESS THAN (TIMESTAMP' 2014-06-09 00:00:00.000000000'),
< repeat PARTITION a bunch more times with incremented DUL2014xx and timestamp dates by 7 days at a time >
PARTITION DULMAXDATE VALUES LESS THAN ( MAXVALUE ) )
ENABLE ROW MOVEMENT PARALLEL )


Immediately after that in the server upgrade log a "AuditOracleDBCreatePartitionConfigureAction complete!" is logged. I can only guess this audit database table was the only database related problem, though it seems that's may not be the case.

The guy who did the ZCM application server upgrade said he ran a ZDC database verify after the upgrade from 11.2.3a to 11.3a & before applying the Win 8.1 and FTF patches, but can't seem to find the report. If we run a zdc database verify now using the downloaded zdc for 11.3a, we get a pretty ugly report (lots of missing tables) - though I can't imagine the system would even start up and (seemingly) work as fine as it is. We found the DB verify logs from BEFORE the upgrade, and they're clean - all tables and procedures/views were present.


Is the ZDC downloaded from the 11.3a files list intended to work with a database verify when the installed release is 11.3 FTF Roll Up 1? How boned are these environments / are they salvageable?