[ASTERIXDB-1945][STO] Cleanup Buffer Cache API
- user model changes: no
- storage format changes: no
- interface changes: yes
INcApplicationContext
- removed IFileMapProvider getFileMapManager();
to hide FileMapManager from other components;
IStorageManager
- IFileMapProvider getFileMapProvider(INCServiceContext ctx);
to hide FileMapManager from other components;
IFileHandle
- added FileReference getFileReference();
to avoid unnecessary casts;
IIOManager
- public void deleteWorkspaceFiles() throws
HyracksDataException;
added throws;
ILSMIndexFileManager
- void createDirs() throws HyracksDataException;
added throws;
IInvertedIndex
- added void purge() throws HyracksDataException;
a. InvertedIndexes don't implement the ITreeIndex interface.
b. when we deactivate a disk component, we need to purge it so
the buffer cache doesn't go through each page.
c. this need to be revisited, ASTERIXDB-1944
IFileMapManager
- int registerFile(FileReference fileRef) throws
HyracksDataException;
return value added for future reference of the index file
inside BufferCache or VirtualBufferCache;
- FileReference unregisterFile(int fileId) throws
HyracksDataException;
return value added for future refernece of the file;
IBufferCache
- int createFile(FileReference fileRef) throws
HyracksDataException;
return value added for future reference of the index file
inside BufferCache or VirtualBufferCache;
- void deleteFile(int fileId) throws HyracksDataException;
remove the dirty page flag since there's no dirty page;
- int openFile(FileReference fileRef) throws
HyracksDataException;
return value added for future reference of the index file
inside BufferCache or VirtualBufferCache;
- added void deleteFile(FileReference file) throws
HyracksDataException;
we used to have this public methods in both BufferCache
and VirtualBufferCache. Now we lifted it into the interface.
AbstractLSMIndex
- removed protected abstract void
destroyMemoryComponent(ILSMMemoryComponent c)
throws HyracksDataException;
It is because turned out when we deactivate, we actually
destroy them. However, because of the not well defined API,
double destroy was okay and so we used to do double destroy.
Details:
This change fixes the buffer cache to follow the API such that:
1. createFile creates the file.
2. deleteFile deletes the file.
3. openFile opens the file.
4. closeFile closes the file.
5. creates existing file is not allowed.
6. deletes deleted file is not allowed.
7. open non existing file is not allowed.
In addition, we hide the file map from all other components.
Change-Id: I0a973c2adb2e7fdcbbf18c7b888af3de5f0acc74
Reviewed-on: https://asterix-gerrit.ics.uci.edu/1843
Tested-by: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
BAD: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Integration-Tests: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Reviewed-by: Till Westmann <tillw@apache.org>
diff --git a/hyracks-fullstack/hyracks/hyracks-api/pom.xml b/hyracks-fullstack/hyracks/hyracks-api/pom.xml
index ddba2d8..f5565fb 100644
--- a/hyracks-fullstack/hyracks/hyracks-api/pom.xml
+++ b/hyracks-fullstack/hyracks/hyracks-api/pom.xml
@@ -102,5 +102,9 @@
<groupId>org.apache.commons</groupId>
<artifactId>commons-collections4</artifactId>
</dependency>
+ <dependency>
+ <groupId>commons-io</groupId>
+ <artifactId>commons-io</artifactId>
+ </dependency>
</dependencies>
</project>
diff --git a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/exceptions/ErrorCode.java b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/exceptions/ErrorCode.java
index 8f36fcd..8a91547 100644
--- a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/exceptions/ErrorCode.java
+++ b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/exceptions/ErrorCode.java
@@ -63,7 +63,7 @@
public static final int FILE_IS_NOT_DIRECTORY = 27;
public static final int CANNOT_READ_FILE = 28;
public static final int UNIDENTIFIED_IO_ERROR_READING_FILE = 29;
- public static final int FILE_DOES_NOT_EXISTS = 30;
+ public static final int FILE_DOES_NOT_EXIST = 30;
public static final int UNIDENTIFIED_IO_ERROR_DELETING_DIR = 31;
public static final int RESULT_NO_RECORD = 32;
public static final int DUPLICATE_KEY = 33;
@@ -100,6 +100,22 @@
public static final int FLUSH_ON_CLOSED_WRITER = 64;
public static final int FAIL_ON_FAILED_WRITER = 65;
public static final int MISSED_FAIL_CALL = 66;
+ public static final int CANNOT_CREATE_FILE = 67;
+ public static final int NO_MAPPING_FOR_FILE_ID = 68;
+ public static final int NO_MAPPING_FOR_FILENAME = 69;
+ public static final int CANNOT_GET_NUMBER_OF_ELEMENT_FROM_INACTIVE_FILTER = 70;
+ public static final int CANNOT_CREATE_BLOOM_FILTER_BUILDER_FOR_INACTIVE_FILTER = 71;
+ public static final int CANNOT_CREATE_BLOOM_FILTER_WITH_NUMBER_OF_PAGES = 72;
+ public static final int CANNOT_ADD_TUPLES_TO_DUMMY_BLOOM_FILTER = 73;
+ public static final int CANNOT_CREATE_ACTIVE_BLOOM_FILTER = 74;
+ public static final int CANNOT_DEACTIVATE_INACTIVE_BLOOM_FILTER = 75;
+ public static final int CANNOT_DESTROY_ACTIVE_BLOOM_FILTER = 76;
+ public static final int CANNOT_PURGE_ACTIVE_INDEX = 77;
+ public static final int CANNOT_PURGE_ACTIVE_BLOOM_FILTER = 78;
+ public static final int CANNOT_BULK_LOAD_NON_EMPTY_TREE = 79;
+ public static final int CANNOT_CREATE_EXISTING_INDEX = 80;
+ public static final int FILE_ALREADY_MAPPED = 81;
+ public static final int FILE_ALREADY_EXISTS = 82;
// Compilation error codes.
public static final int RULECOLLECTION_NOT_INSTANCE_OF_LIST = 10000;
diff --git a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/FileReference.java b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/FileReference.java
index 12a138a..b48ba14 100644
--- a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/FileReference.java
+++ b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/FileReference.java
@@ -56,12 +56,12 @@
if (!(o instanceof FileReference)) {
return false;
}
- return file.equals(((FileReference) o).file);
+ return path.equals(((FileReference) o).path) && dev.equals(((FileReference) o).dev);
}
@Override
public int hashCode() {
- return file.hashCode();
+ return path.hashCode();
}
/**
diff --git a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/IFileHandle.java b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/IFileHandle.java
index a24c667..768ab87 100644
--- a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/IFileHandle.java
+++ b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/IFileHandle.java
@@ -18,9 +18,8 @@
*/
package org.apache.hyracks.api.io;
-/**
- * IFileHandle is an interface that exists only for Java compilers to perform static typing
- * when handling file handle objects. Users must not implement this interface.
- */
+@FunctionalInterface
public interface IFileHandle {
+
+ FileReference getFileReference();
}
diff --git a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/IIOManager.java b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/IIOManager.java
index 50ef4f0..b0cc07a 100644
--- a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/IIOManager.java
+++ b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/io/IIOManager.java
@@ -59,7 +59,7 @@
public long getSize(IFileHandle fileHandle);
- public void deleteWorkspaceFiles();
+ public void deleteWorkspaceFiles() throws HyracksDataException;
/**
* @param ioDeviceId
@@ -79,6 +79,7 @@
/**
* Gets a file reference from an absolute path
+ *
* @deprecated
* use getFileRef(int ioDeviceId, String path) instead
* @param path
@@ -90,6 +91,7 @@
/**
* Create a workspace file with the given prefix
+ *
* @param prefix
* @return A FileReference for the created workspace file
* @throws HyracksDataException
diff --git a/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/util/IoUtil.java b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/util/IoUtil.java
new file mode 100644
index 0000000..f4c5114
--- /dev/null
+++ b/hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/util/IoUtil.java
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.hyracks.api.util;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.hyracks.api.exceptions.ErrorCode;
+import org.apache.hyracks.api.exceptions.HyracksDataException;
+import org.apache.hyracks.api.io.FileReference;
+
+/**
+ * This util class takes care of creation and deletion of files and directories
+ * and throws the appropriate error in case of failure.
+ */
+public class IoUtil {
+
+ private IoUtil() {
+ }
+
+ /**
+ * Delete a file
+ *
+ * @param fileRef
+ * the file to be deleted
+ * @throws HyracksDataException
+ * if the file doesn't exist or if it couldn't be deleted
+ */
+ public static void delete(FileReference fileRef) throws HyracksDataException {
+ delete(fileRef.getFile());
+ }
+
+ /**
+ * Delete a file
+ *
+ * @param file
+ * the file to be deleted
+ * @throws HyracksDataException
+ * if the file doesn't exist or if it couldn't be deleted
+ */
+ public static void delete(File file) throws HyracksDataException {
+ try {
+ if (file.isDirectory()) {
+ FileUtils.deleteDirectory(file);
+ } else {
+ Files.delete(file.toPath());
+ }
+ } catch (IOException e) {
+ throw HyracksDataException.create(ErrorCode.CANNOT_DELETE_FILE, e, file.getAbsolutePath());
+ }
+ }
+
+ /**
+ * Create a file on disk
+ *
+ * @param fileRef
+ * the file to create
+ * @throws HyracksDataException
+ * if the file already exists or if it couldn't be created
+ */
+ public static void create(FileReference fileRef) throws HyracksDataException {
+ if (fileRef.getFile().exists()) {
+ throw HyracksDataException.create(ErrorCode.FILE_ALREADY_EXISTS, fileRef.getAbsolutePath());
+ }
+ fileRef.getFile().getParentFile().mkdirs();
+ try {
+ if (!fileRef.getFile().createNewFile()) {
+ throw HyracksDataException.create(ErrorCode.FILE_ALREADY_EXISTS, fileRef.getAbsolutePath());
+ }
+ } catch (IOException e) {
+ throw HyracksDataException.create(ErrorCode.CANNOT_CREATE_FILE, e, fileRef.getAbsolutePath());
+ }
+ }
+}
diff --git a/hyracks-fullstack/hyracks/hyracks-api/src/main/resources/errormsg/en.properties b/hyracks-fullstack/hyracks/hyracks-api/src/main/resources/errormsg/en.properties
index 4bf069c..9f983a7 100644
--- a/hyracks-fullstack/hyracks/hyracks-api/src/main/resources/errormsg/en.properties
+++ b/hyracks-fullstack/hyracks/hyracks-api/src/main/resources/errormsg/en.properties
@@ -46,7 +46,7 @@
27 = File %1$s is not a directory
28 = User doesn't have read permissions on the file %1$s
29 = Unidentified IO error occurred while reading the file %1$s
-30 = File %1$s doesn't exists
+30 = File %1$s doesn't exist
31 = Unidentified IO error occurred while deleting the dir %1$s
32 = No record for partition %1$s of result set %2$s
33 = Inserting duplicate keys into the primary storage
@@ -83,5 +83,21 @@
64 = Data pipeline protocol violation: flush() is called on a closed writer
65 = Data pipeline protocol violation: fail() is called twice on a writer
66 = Data pipeline protocol violation: fail() is not called by the upstream when there is a failure in the downstream
+67 = Cannot create the file: %1$s
+68 = No mapping found for file id %1$s
+69 = No mapping found for filename %1$s
+70 = Cannot get the number of elements from inactive bloom filter
+71 = Failed to create the bloom filter builder since it is not active
+72 = Cannot create a bloom filter with number of pages = %1$s
+73 = Cannot add elements to this filter since it is supposed to be empty (number of elements hint passed to the filter during construction was 0)
+74 = Failed to create the bloom filter since it is active
+75 = Failed to deactivate the bloom filter since it is inactive
+76 = Failed to destroy the bloom filter since it is active
+77 = Failed to purge the index since it is active
+78 = Failed to purge the bloom filter since it is active
+79 = Cannot bulk-load a non-empty tree
+80 = Cannot create index because it already exists
+81 = File %1$s is already mapped
+82 = Failed to create the file %1$s because it already exists
10000 = The given rule collection %1$s is not an instance of the List class.