Puppetdb missing replace_catalogue commands

asked 2017-04-05 23:10:19 -0500

A few times we observed that replace_catalogue command issued from puppetmaser hasn't been received (or received but not processed) by puppetdb. We couldn't yet figure out what is going and this is NOT something that happens frequently. We can clearly see from the puppetmaster log that the command is issued (for multiple nodes), but the puppetdb log indicates that only some of them are processed. We are using puppet 3.6.2 and puppetdb 2.3.8.

We found the below exception in puppetdb log when this happened, but we aren't sure if that is related as we don't recall seeing the same exception last time when we encountered same issue.

2017-04-03 14:49:28,876 WARN [o.a.a.b.T.Service] Async error occurred: java.lang.IllegalStateException: PageFile is not loaded java.lang.IllegalStateException: PageFile is not loaded at org.apache.kahadb.page.PageFile.assertLoaded(PageFile.java:800) ~[puppetdb.jar:na] at org.apache.kahadb.page.PageFile.tx(PageFile.java:294) ~[puppetdb.jar:na] at org.apache.activemq.broker.scheduler.JobSchedulerImpl.schedule(JobSchedulerImpl.java:110) ~[puppetdb.jar:na] at org.apache.activemq.broker.scheduler.SchedulerBroker.send(SchedulerBroker.java:185) ~[puppetdb.jar:na] at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129) ~[puppetdb.jar:na] at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96) ~[puppetdb.jar:na] at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:306) ~[puppetdb.jar:na] at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135) ~[puppetdb.jar:na] at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:453) ~[puppetdb.jar:na] at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:681) ~[puppetdb.jar:na] at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292) ~[puppetdb.jar:na] at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:150) [puppetdb.jar:na] at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) [puppetdb.jar:na] at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) [puppetdb.jar:na] at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:231) [puppetdb.jar:na] at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) [puppetdb.jar:na] at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) [puppetdb.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0131] at java.lang.Thread.run(Thread.java:745) [na:1.7.0131] 2017-04-03 14:49:28,877 WARN [o.s.j.c.CachingConnectionFactory] Encountered a JMSException - resetting the underlying JMS Connection javax.jms.JMSException: PageFile is not loaded at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) ~[puppetdb.jar:na] at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1934) ~[puppetdb.jar:na] at org.apache.activemq.ActiveMQConnection$3$1.run(ActiveMQConnection.java:1855) ~[puppetdb.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java ...

(more)
edit retag flag offensive close merge delete