pathname in datastore cooked files

[dealharris]

Adding additional shard services to a current precog instance requires copying over the shard-data path. However, since the original pathname is stored in the cooked files, the new shard service will not be able to query the new (copied) shard-data path. Here is the config:

queryExecutor {
systemId = "dev2"
precog {
storage {
root = /home/precog/work/shard2-data/
}
}

Here is the error log:

2013-10-17 12:05:15,061 [patcher-13] E c.p.s.s.SyncQueryServiceHandler {} - Error executing shard query:
java.io.FileNotFoundException: /home/precog/work/shard-data/data/0000000066/statistics/jobs/jobgroups/perAuthProjections/b35a170be4ef15e078a741233a0f4245854c9bb3/cooked_blocks/segment-0--580161124-Decimal6211499993968270611.cooked (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:137)
at com.precog.niflheim.CookedReader.com$precog$niflheim$CookedReader$$read(CookedReader.scala:32)
at com.precog.niflheim.CookedReader$$anonfun$load$1$$anonfun$apply$15$$anonfun$9.apply(CookedReader.scala:119)
at com.precog.niflheim.CookedReader$$anonfun$load$1$$anonfun$apply$15$$anonfun$9.apply(CookedReader.scala:118)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
at scala.collection.immutable.List.foreach(List.scala:76)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
at scala.collection.immutable.List.map(List.scala:76)
at com.precog.niflheim.CookedReader$$anonfun$load$1$$anonfun$apply$15.apply(CookedReader.scala:118)
at com.precog.niflheim.CookedReader$$anonfun$load$1$$anonfun$apply$15.apply(CookedReader.scala:117)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:233)
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
at scala.collection.immutable.List.foreach(List.scala:76)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:233)
at scala.collection.immutable.List.map(List.scala:76)
at com.precog.niflheim.CookedReader$$anonfun$load$1.apply(CookedReader.scala:117)
at com.precog.niflheim.CookedReader$$anonfun$load$1.apply(CookedReader.scala:116)
at scalaz.Validation$class.flatMap(Validation.scala:141)
at scalaz.Success.flatMap(Validation.scala:329)
at com.precog.niflheim.CookedReader.load(CookedReader.scala:116)
at com.precog.niflheim.CookedReader$$anonfun$6.apply(CookedReader.scala:75)
at com.precog.niflheim.CookedReader$$anonfun$6.apply(CookedReader.scala:74)
at scala.Option.map(Option.scala:133)
at com.precog.niflheim.CookedReader.snapshotRef(CookedReader.scala:74)
at com.precog.niflheim.NIHDBSnapshot$$anonfun$getBlockAfter$1.apply(NIHDBSnapshot.scala:54)
at com.precog.niflheim.NIHDBSnapshot$$anonfun$getBlockAfter$1.apply(NIHDBSnapshot.scala:53)
at scala.Option.map(Option.scala:133)
at com.precog.niflheim.NIHDBSnapshot$class.getBlockAfter(NIHDBSnapshot.scala:53)
at com.precog.niflheim.NIHDBSnapshot$$anon$1.getBlockAfter(NIHDBSnapshot.scala:18)
at com.precog.niflheim.NIHDB$$anonfun$getBlockAfter$1.apply(NIHDBActor.scala:75)
at com.precog.niflheim.NIHDB$$anonfun$getBlockAfter$1.apply(NIHDBActor.scala:75)
at akka.dispatch.Future$$anonfun$map$1.liftedTree3$1(Future.scala:625)
at akka.dispatch.Future$$anonfun$map$1.apply(Future.scala:624)
at akka.dispatch.Future$$anonfun$map$1.apply(Future.scala:621)
at akka.dispatch.DefaultPromise.akka$dispatch$DefaultPromise$$notifyCompleted(Future.scala:943)
at akka.dispatch.DefaultPromise$$anonfun$tryComplete$1$$anonfun$apply$mcV$sp$4.apply(Future.scala:920)
at akka.dispatch.DefaultPromise$$anonfun$tryComplete$1$$anonfun$apply$mcV$sp$4.apply(Future.scala:920)
at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
at scala.collection.immutable.List.foreach(List.scala:76)
at akka.dispatch.DefaultPromise$$anonfun$tryComplete$1.apply$mcV$sp(Future.scala:920)
at akka.dispatch.Future$$anon$4$$anonfun$run$1.apply$mcV$sp(Future.scala:386)
at akka.dispatch.Future$$anon$4$$anonfun$run$1.apply(Future.scala:378)
at akka.dispatch.Future$$anon$4$$anonfun$run$1.apply(Future.scala:378)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
at akka.dispatch.Future$$anon$4.run(Future.scala:378)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:94)
at akka.jsr166y.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1381)
at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

[jdegoes]

This is a bug, we should find a way to avoid storing path information inside files.