| commit | 9340d8c44d594829c1761700c24436b70ec9241a | [log] [tgz] |
|---|---|---|
| author | Hussain Towaileb <Hussain.Towaileb@couchbase.com> | Mon Apr 08 20:51:02 2024 +0300 |
| committer | Hussain Towaileb <hussainht@gmail.com> | Thu Apr 18 14:41:02 2024 +0000 |
| tree | fa78c5893dece98b5293f916eae3ae8ad457fc5d | |
| parent | 29f1d423b7a8ebff0abc7ff40534463a66b1a33b [diff] |
Avoid writing empty files when COPYing TO S3 Change-Id: Iee18cb7458495471fe38e5f6f2f3721d8ce9c12f Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18230 Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu> Reviewed-by: Hussain Towaileb <hussainht@gmail.com>
diff --git a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java index f9f98da..95dc962 100644 --- a/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java +++ b/asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/writer/ExternalFileWriter.java
@@ -57,11 +57,17 @@ // Ignore writing values for unresolvable partition paths return; } - writer.write(value); - tupleCounter++; + + // create a new file only when we reach the maximum tuples and we know a new tuple is incoming + // e.g., if max is 1000, we hit tuple 1001, we will upload and create a new file, if we only have 1000 + // we will stop here, and calling the close/finish will upload whatever is written. This is to avoid + // creating and uploading empty files if (tupleCounter >= maxResultPerFile) { newFile(); } + + writer.write(value); + tupleCounter++; } @Override