Tags: microsoft/garnet
Tags
LogRecord and SpanByte changes (#1186) * Refactors prior to storage-v2: - Move CountdownWrapper to its own file - Rename Lockable* to Transactional* - Rename Transient* to Ephemeral* * WIP; first cut at LogRecord etc. implementations * WIP: more LogRecord * More *LogRecord, and DiskRecord * WIP: Convert allocators to new LogRecord and DiskRecord * WIP: incorporate LogRecord into ISessionFunctions * More logRecord WIP * More WIP on MainSessionFunctions RMWMethods (and PrivateMethods) with LogRecord * WIP LogRecord (MainStore RMWMethods) * WIP: Consolidate to a single LogRecord; convert InternalRMW * TKey removal * A few more TKey removals * WIP more logRecord * Remove pendingConterxt.recordInfo as it is no longer in RecordMetadata Other work to get CreateNewRecordRMW clean * WIP; Internal(RUMD) converted. * added PendingContext implementation of ISourceLogRecord * Finish ContinuePending * Iterator changes - consolidate to RecordScanIterator * Mostly IO work * WIP: Reintroduce TValue through ObjectAllocator, LogRecord, etc. * WIP Convert Garnet to LogRecord and ObjectAllocator (MainStoreFunctions first pass done) * WIP converting Garnet to non-ref SpanByte-only keys * More WIP on Garnet migration to LogRecord * WIP Tsavorite LogRecord odds and ends * Rename TsavoriteLog to TsavoriteAof * More processing-layer LogRecord etc. fixes * fix iterator as ISourceLogRecord * More Expiration etc. cleanup in processing layer * WIP BlockAllocate work * FInalize removal of SpanByte.Metadata * More param fixes in processing layer * SpanField and OptionalFieldsShift structs * Propagate some SpanField etc. changes * First pass at LogRecord-based Revivification * Finish OverflowAllocator * OK now I finished this iteration of OverflowAllocator * improve callers' use of OverflowAllocator * More LogRecord-related cleanup * // TODOMigrate: temporarily clear errors related to Expiration in Key Migration * Fixes for first test runs * More fixes for CopyRecord and setting of Expiration etc * Cleanup OversizeAllocator.BlockHeader and usage * SpanField and LogRecord documentation and cleanup; BlockHeader GetUserSize * More improvements to overflow allocator and usage * More fixes for tests * Remove unneeded IAllocator(Callbacks) methods (superseded by LogRecord) * more overflow allocator fixes * sizeInfo on ISessionFunctions and TrySetValueSpanLength improvements * Remove SpanByte serialized usage (but leave an assert on .Serialized in case I missed some). Also fix CopyUpdater copies from old record and some more OverflowAllocator tweaks * Improved FreeList handling and ObjectIdMap growth/FreeList via SimpleConcurrentStack and TailAndLatch * Make sure DiskLogRecord Value and Record lengths are long * Remove ISerializer (not used and not planned to be) * TIghten up SpanByte !IsSerialized assert and fix SETEX "expiration set mismatch" * Some TODO cleanup * Begin converting Tsavorite tests and benchmarks to LogRecord * Tsavorite RevivificationTests and ObjectTests * More ObjectIdMap and OverflowAllocator work and tests * More overflowAllocator work: FixedSizePages and other fixes for YCSB, add tests and fixes for OversizePages * Clean up Locking.md (had some referenced to obsolete locking modes); start of LogRecord-related docs * support for "inline struct" as SpanByte for Object value optimization * finish "inline struct" testing; update processing layer to Tsavorite changes; update LogRecord docs * Convert RecordFieldInfo Key/Value TotalSize to DataSize - caller should not need to know about the length prefix (especially bc it changes between Inline, Overflow, ObjectId) * Fixes from merging main * More testing and fixes * More merge fixes * fix more tests * minor comment cleanup * some doc-related updates, mostly comments * Add DisposeReason.Deleted, .Elided Add YCSB --di (Delete and reInsert) option * WIP Span<byte> * WIP Span<byte> * WIP Span<byte> * WIP Span<byte> * WIP Span<byte> * WIP Span<byte> * WIP Span<byte> * WIP Span<byte> (Tsavorite builds, with a couple hacks) * WIP Span<byte>: YCSB fixes * WIP Span<byte>: more fixes found during YCSB * Switching CTT, CTRC, TempKv insertions to TSourceLogRecord direct copies (eliminating need for ISessionFunctions calls for those operations) * WIP Span<byte>: more cleanup in Compaction (no TSessionFunctions needed), other fixes related to prior changes * WIP Span<byte>: LogRecord revisions in functions and prep Migrate/Repl for change to LogRecord based * ISessionFunctions revisions - Rename SingleWriter() => InitialWriter(), ConcurrentWriter() => InPlaceWriter() - Rename SinglecDeleter() => InitialDeleter(), ConcurrentDeleter() => InPlaceDeleter() - Consolidate SingleReader() and ConcurrentReader() to just Reader() * Remove TInput, TOutput, TContext from Compact and PrepareIOForConditionalOperation chain * Remove WriteReason as it is no longer needed with Compaction, CopyToTail, and CopyToReadCache now being fully internal operations. * Missed a few WriteReason * WIP Span<byte>: Cleanup DiskLogRecord deserialization and remove some no-longer-needed allocator functions that are now done by DLR * Varebyte in DiskLogRecord * More LogRecord testing * FInish DiskLogRecord serialization tests * Rename SpanField to LogField * Various cleanup, including PrepareForRevivification * WIP on converting Replication and Migration to LogRecord * WIP on Repl (done-ish) and Migrate (mostly done except ISF) conversion to DiskLogRecord * Split IHO.Size into .MemorySize, DiskSize Almost done with Migrate changeover to LogRecord * Finish LogRecordization of Migrate * First fixes for Migration testing * More Migrate testing; use SBAM.MemorySpan * SImple Migrate tests now run * Simple diskless replica runs * Fixes for in-memory tests * Add UpsertValueSelector * Remove PendingContext.IsAsync (a remnant of earlier, since-removed async APIs) from Tsavorite * Cleanup Read initialization of PendingContext * Fix record alignment in YCSB test * tune HashBytes * Fix initial inserting in YCSB * Update LogRecord.md and some cleanup * Restore TsavoriteAof naming back to TsavoriteLog * Add changes for AddressType in logicalAddresses and enable minimal IO for SpanByteAllocator. The AddressType change is a breaking on-disk format change: it shuffles bits around in RecordInfo to add an additional bit adjacent to the old ReadCache bit to mark an address as: - 00: Reserved - 11: ReadCache - 10: InMemory portion of the main log - 01: On-Disk * Fix a few more cursor tests (int => long) * Fixes for review comments (except for "ref RecordSizeInfo") * TSourceLogRecord and RecordSizeInfo parameters are readonly so make them "in" rather than "ref" * Fix some missing ref => in for TSourceLogRecord * Merge fixes; MigrateKeysWithObjects test still fails * Fix Object store migration typo * format * ObjectAllocator serialization - not yet complete * Garnet Storage v2 (#1384) * wip * wip * wip * Added unified store session * Correcting generic typing * Added MEMORY USAGE + TYPE to unified ops * Added TTL, EXPIRETIME and EXISTS to unified ops * implemented DEL in unified ops * wip - expire & persist (broken) * wip - adding expire to unified ops * wip - expire * add cref to server-side replication inter-node commands * fix server-side BeginRecoverReplica * wip * Fix transaction key locking * format * Some test fixes * Fixing tests * reverting a couple of unnecessary changes * Eliminating more multi-context methods from API * Removed some unnecessary stuff * Some more cleanup to TransactionManager * merge tedhar/storage-v2 (ObjectAllocator serialization updates) * Updating memory usage values * format * Handling wrong type ops * Revert "Updating memory usage values" This reverts commit 88ba307. * fix no-object-log case * Fixes for Tsavorite UTs * Fixes for Tsavorite UTs (mostly ReadCache, TsavoriteLog, Compaction) * Tsavorite Iterator work and UT re-enabling * Fixes to Object iteration, LogRecord.ToString() * Add RecordMetadata.ETag * Readding --no-obj config * fix * test fix * Prep for Recovery * wip * wip * fix * More fixes for UT (mostly Recovery, Migration) * Moving DELIFEXPIM to unified store * ObjectLogTail in Recovery, and more UT fixes (Migration record serialization, ReadCache size and tailAddress verification calculations, etc.) * More Tsavorite recovery tests * Removing unnecessary isObject flag from record serialization --------- Co-authored-by: Vasileios Zois <[email protected]> Co-authored-by: TedHartMS <[email protected]> * Removing unnecessary output flag * Fixes for StateMachineTests (Recovery) * Fixes, mostly for Replication to include object log for UnifiedStore; correctly initialized SegmentSize for ObjectLogDevice; other fixes to UTs to work with UnifiedStore * fixes * Fixes for RESP UTs for Unified; clear logRecord on deserialization error * Remove SerializedSize(IsExact) from IHeapObject; various UT fixes * All Cluster.test now passes * small fixes * Fix TailAddress from Rescovery not being PageHeader-aligned * Move RENAME to Unified Store and modify for LogRecord * Fixing rename with etag + removing remaining unnecessary GarnetObjectTypes * Add ITsavoriteRecordObserver and fix Readcache size tracking * Convert LogRecord to use RecordLength * More work on LogRecord.RecordLength * Finish LogRecord.RecordLength * Remove ObjectDb* from Sample*Txn * MultiLevelPageArray stress fixes; improved LogRecord and RecordDataHeader ToString() * Fix GetObjectLogSegmentSize() * update docs * Fix a couple comments * Trim ObjectIdMap arrays on ClearPage * wip * wip * wip * wip * wip * wip * wip * wip * formatting tsavo * Fixing website build error * some renames * small rename * some small changes * Version update * Address review comments; implement OA Truncate; add --value-overflow-threshold configuration option * ObjectAllocator testing * Add large (multi-GB) object tests (and fix associated issues, mostly object-log segments); PendingContext is no longer itself a TSourceLogRecord; * readcache fixes * Move KVSettings from GarnetServer to the individual Database, so we can dispose the log devices. Remove KVSettings from CachedSizeTracker as it was unused Fix some Tsavorite LogRecord-lifetime issues on scan and conditional insert/push * WIP to fix replica for object log segments * Fixes for ObjectLog recovery * Some additional ObjectRecovery test conversion, not ready yet * Convert and re-enable more Tsavorite ObjectRecovery tests. Fix CacheSerializedData to not overwrite if CopyUpdater already set the object Fix foldover checkpoint storage of object log tail * format * A couple fixes for DiskLogRecord serialization * Fix SimpleIncrementByFloatWithOutOfRangeFloat exception verification to use Throws.TypeOf<>().Or.TypeOf<>()) Make SpanByteRcMultiThreadTest [Explicit] until I figure out the request_key issue * Address review comments Fix merge error that resulted in Cluster timeouts Minor cleanup * format * Comment out [Repeat] --------- Co-authored-by: Tal Zaccai <[email protected]> Co-authored-by: Vasileios Zois <[email protected]>
fix txn replay for fuzzy region (#1412) * fix txn replay for fuzzy region * renaming and comments * fix test waiting logic * add aof replay coordinator * augment tests * cleanup and comments * fix formatting * cleanup * revert dispose at recover to avoid breaking tests * adding comments and minor cleanup * addressing comments * ensure txnManager.Commit on replay does not write to replica AOF * assert TxnManager Aof disabled when replaying * ensure read-only txn at replica do not write into the AOF * skip writes to the AOF transaction contains only read operations * test fix * bump version
Fix: snapshot checkpoint flush corner case (#1413) * Fix: snapshot checkpoint flush corner case * nit * Ensure serialization completion before nullifying. Introduce a loop to check `serializationState` and ensure that any concurrent serialization of `oldValue` is completed before setting it to `null` and returning. This prevents concurrent modification of the shallow-copied object contents. * misc bugfixes * update version
Resetting replication recovery when cluster reset is issued (#1319) * Resetting replication recovery when cluster reset is issued * validating using clusterendpoint during reset * missed flag change * using reset cancellation token * Merge fix * ADded other recovery statuses during reset recovery. * using storeWrapper.serverOptions.ReplicaAttachTimeout for replica diskless sync * ran dotnet formatter * added reset during replication tests * ran dotnet format * removed if debug * if debug encompasses using statements * version bump --------- Co-authored-by: Tejas Kulkarni <[email protected]> Co-authored-by: Vasileios Zois <[email protected]>
PreviousNext