Now that I have the DB purge tool running, I started by purging 21
Collections, and it worked. (Amusingly DB under the covers went from
29GB to 35GB after that purge. I assume my DBA needs to compact after I
am done).

Then I tried to do the 65 snapshots (By doing everything older than 10
days, *) and it errored on the console as:

Unexpected response code received: 489
Error code from server: InternalExceptionOccured
Error message from server: Encountered unexpected error.

In catalina, I found:

[SEVERE] 2018-08-21 08:12:23
com.netiq.iac.common.logging.IACLoggingUtils logExceptionError -
[IG-SERVER] Encountered unexpected error: could not execute statement
com.netiq.common.i18n.LocalizedException: Encountered unexpected error:
could not execute statement
at com.netiq.iac.common.IacException.<init>(IacExcept ion.java:105)
at
com.netiq.iac.persistence.spi.exception.IacPersist enceException.<init>(IacPersistenceException.java: 118)
at
com.netiq.iac.persistence.dao.DataTransactor.execu te(DataTransactor.java:133)
at
com.netiq.iac.persistence.dao.ops.SnapshotDeletion .deleteDataProductions(SnapshotDeletion.java:1189)
at
com.netiq.iac.persistence.dao.ops.SnapshotDeletion $8.eval(SnapshotDeletion.java:532)
at
com.netiq.iac.persistence.dao.ops.SnapshotDeletion $8.eval(SnapshotDeletion.java:1)
at
com.netiq.iac.persistence.dao.ops.ResultTransactor .execute(ResultTransactor.java:113)
at
com.netiq.iac.persistence.dao.ops.SnapshotDeletion .deleteSnapshot(SnapshotDeletion.java:541)
at
com.netiq.iac.persistence.dao.ops.SnapshotDeletion .access$6(SnapshotDeletion.java:494)
at
com.netiq.iac.persistence.dao.ops.SnapshotDeletion $5.eval(SnapshotDeletion.java:450)
at
com.netiq.iac.persistence.dao.ops.SnapshotDeletion $5.eval(SnapshotDeletion.java:1)
at
com.netiq.iac.persistence.dao.ops.ResultTransactor .execute(ResultTransactor.java:113)
at
com.netiq.iac.persistence.dao.ops.SnapshotDeletion .deleteSnapshot(SnapshotDeletion.java:457)
at
com.netiq.iac.persistence.service.SnapshotPurgeSer vice$3.doIn(SnapshotPurgeService.java:107)
at
com.netiq.iac.persistence.dao.DataTransactor.execu te(DataTransactor.java:121)
at
com.netiq.iac.persistence.service.SnapshotPurgeSer vice.purgeEntities(SnapshotPurgeService.java:112)
at
com.netiq.iac.server.rest.DataManagementService.pu rgeEntities(DataManagementService.java:378)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.sun.jersey.spi.container.JavaMethodInvokerFact ory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.A bstractResourceMethodDispatchProvider$ResponseOutI nvoker._dispatch(AbstractResourceMethodDispatchPro vider.java:205)
at
com.sun.jersey.server.impl.model.method.dispatch.R esourceJavaMethodDispatcher.dispatch(ResourceJavaM ethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRul e.accept(HttpMethodRule.java:302)
at
com.sun.jersey.server.impl.uri.rules.RightHandPath Rule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.ResourceClass Rule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPath Rule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceC lassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicat ionImpl._handleRequest(WebApplicationImpl.java:154 2)
at
com.sun.jersey.server.impl.application.WebApplicat ionImpl._handleRequest(WebApplicationImpl.java:147 3)
at
com.sun.jersey.server.impl.application.WebApplicat ionImpl.handleRequest(WebApplicationImpl.java:1419 )
at
com.sun.jersey.server.impl.application.WebApplicat ionImpl.handleRequest(WebApplicationImpl.java:1409 )
at
com.sun.jersey.spi.container.servlet.WebComponent. service(WebComponent.java:409)
at
com.sun.jersey.spi.container.servlet.ServletContai ner.service(ServletContainer.java:558)
at
com.sun.jersey.spi.container.servlet.ServletContai ner.service(ServletContainer.java:733)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:742)
at
org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:231)
at
org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:166)
at
org.apache.tomcat.websocket.server.WsFilter.doFilt er(WsFilter.java:52)
at
org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:166)
at
com.netiq.iac.common.j2ee.NoCacheFilter.doFilter(N oCacheFilter.java:65)
at
org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:166)
at
com.netiq.common.i18n.impl.BestLocaleChooserFilter .doFilter(BestLocaleChooserFilter.java:234)
at
org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:166)
at
com.netiq.iac.server.common.audit.AuditLogFilter.d oFilter(AuditLogFilter.java:134)
at
org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:166)
at
com.netiq.iac.server.j2ee.RestApiAuthFilter.doFilt er(RestApiAuthFilter.java:145)
at
org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:166)
at
com.netiq.iac.server.j2ee.AuthFilter.doFilter(Auth Filter.java:239)
at
org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:166)
at
com.netiq.iac.server.j2ee.CORSFilter.doFilter(CORS Filter.java:80)
at
org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:166)
at
org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:199)
at
org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:96)
at
org.apache.catalina.authenticator.AuthenticatorBas e.invoke(AuthenticatorBase.java:478)
at
org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:140)
at
org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:81)
at
org.apache.catalina.valves.AbstractAccessLogValve. invoke(AbstractAccessLogValve.java:650)
at
org.apache.catalina.core.StandardEngineValve.invok e(StandardEngineValve.java:87)
at
org.apache.catalina.connector.CoyoteAdapter.servic e(CoyoteAdapter.java:342)
at
org.apache.coyote.http11.Http11Processor.service(H ttp11Processor.java:803)
at
org.apache.coyote.AbstractProcessorLight.process(A bstractProcessorLight.java:66)
at
org.apache.coyote.AbstractProtocol$ConnectionHandl er.process(AbstractProtocol.java:868)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProce ssor.doRun(NioEndpoint.java:1459)
at
org.apache.tomcat.util.net.SocketProcessorBase.run (SocketProcessorBase.java:49)
at
java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$Wrapping Runnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.exception.SQLGrammarException: could not
execute statement
at
org.hibernate.exception.internal.SQLStateConversio nDelegate.convert(SQLStateConversionDelegate.java: 123)
at
org.hibernate.exception.internal.StandardSQLExcept ionConverter.convert(StandardSQLExceptionConverter .java:49)
at
org.hibernate.engine.jdbc.spi.SqlExceptionHelper.c onvert(SqlExceptionHelper.java:126)
at
org.hibernate.engine.jdbc.spi.SqlExceptionHelper.c onvert(SqlExceptionHelper.java:112)
at
org.hibernate.engine.jdbc.internal.ResultSetReturn Impl.executeUpdate(ResultSetReturnImpl.java:211)
at
org.hibernate.engine.query.spi.NativeSQLQueryPlan. performExecuteUpdate(NativeSQLQueryPlan.java:211)
at
org.hibernate.internal.SessionImpl.executeNativeUp date(SessionImpl.java:1346)
at
org.hibernate.internal.SQLQueryImpl.executeUpdate( SQLQueryImpl.java:389)
at
com.netiq.iac.persistence.dao.ops.SnapshotDeletion $18.doIn(SnapshotDeletion.java:1175)
at
com.netiq.iac.persistence.dao.DataTransactor.execu te(DataTransactor.java:121)
... 74 more
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect
syntax near ')'.
at
com.microsoft.sqlserver.jdbc.SQLServerException.ma keFromDatabaseError(SQLServerException.java:259)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.ge tNextResult(SQLServerStatement.java:1547)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStat ement.doExecutePreparedStatement(SQLServerPrepared Statement.java:548)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStat ement$PrepStmtExecCmd.doExecute(SQLServerPreparedS tatement.java:479)
at
com.microsoft.sqlserver.jdbc.TDSCommand.execute(IO Buffer.java:7344)
at
com.microsoft.sqlserver.jdbc.SQLServerConnection.e xecuteCommand(SQLServerConnection.java:2713)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.ex ecuteCommand(SQLServerStatement.java:224)
at
com.microsoft.sqlserver.jdbc.SQLServerStatement.ex ecuteStatement(SQLServerStatement.java:204)
at
com.microsoft.sqlserver.jdbc.SQLServerPreparedStat ement.executeUpdate(SQLServerPreparedStatement.jav a:426)
at
org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedSta tement.executeUpdate(DelegatingPreparedStatement.j ava:97)
at
org.apache.tomcat.dbcp.dbcp2.DelegatingPreparedSta tement.executeUpdate(DelegatingPreparedStatement.j ava:97)
at
org.hibernate.engine.jdbc.internal.ResultSetReturn Impl.executeUpdate(ResultSetReturnImpl.java:208)
... 79 more


And then the response to the IAC CLient:

[SEVERE] 2018-08-21 08:15:52
com.netiq.iac.common.logging.IACLoggingUtils logExceptionError -
[IG-SERVER] Encountered unexpected error: could not execute statement
[FINE] 2018-08-21 08:15:52
com.netiq.iac.server.common.audit.AuditLogFilter logRequestOutcome -
[IG-SERVER] API server
request{"sender":"129.106.158.23","user":{"userNam e":"Geoffrey
Carman","userId":"gcarman","uniqueUserId":"dfea2b8 c40394db293eeaa7ef9066fc4"},"body":"[EMPTY]","headers":{"host":"idgovidlv001.uth.tmc.edu:8443 ","content-type":"application/json","connection":"Keep-Alive","accept-encoding":"gzip,deflate","accept":"application/json","user-agent":"Apache-HttpClient/UNAVAILABLE
(java
1.5)"},"params":{"retentionTime":"1533963600041"}, "path":"/api/data/mgt/purgeable/SNAPSHOT","method":"DELETE"}
API server
response{"body":"{\"Fault\":{\"Code\":{\"Value\":\ "Sender\",\"Subcode\":{\"Value\":\"InternalExcepti onOccured\"}},\"Reason\":{\"Text\":\"Encountered
unexpected error.\"}}}","status":200}


Alas, it does not tell me which SQL statement was bad?