VERIFY: yql/essentials/minikql/mkql_mem_info.cpp:137 VerifyDebug(): requirement GetUsage() == 0 failed

[naspirato]

Version: 1bd8df7 , builded ./ya make -j128 --build=release ydb/apps/ydbd
During workload log column on cluster LONG

все корки

https://coredumps.yandex-team.ru/core?core_hash=636989760691639598&itype=kikimr
https://coredumps.yandex-team.ru/core?core_hash=918307898284509845&itype=kikimr
https://coredumps.yandex-team.ru/core?core_hash=1076196737967287621&itype=kikimr

нужен стейт кластера
Как получаем стейт

0) byuld ydb cli from main
1)
./ydb --endpoint grpc://vla5-2566.search.yandex.net:2135 --database /Root/db1 workload log init --store column --ttl 240 --path log_workload_select_column --len 1000 --key-cols 18 --int-cols 18 --min-partitions 100 --partition-size 10 --auto-partition 0;
2) параллельно запустить
./ydb --verbose --endpoint grpc://vla5-2566.search.yandex.net:2135 --database /Root/db1 workload log run select  --path log_workload_column --client-timeout 18000000 --seconds 10000
./ydb --verbose --endpoint grpc://vla5-2566.search.yandex.net:2135 --database /Root/db1 workload log run bulk_upsert  --path log_workload_column --len 1000 --key-cols 18 --int-cols 18 --timestamp_deviation 180 --seconds 10000 --threads 100

и когда ноды начнут падать мы увидим

при потоке запросов на bulk_upsert + select начинают падать запросы т.к. выключаются ноды

если выключить поток запросов - ноды поднимутся

после глобального падения нод есть развилка

• ноды могуть упасть на единичном запросе, например выполнить вот этот запрос в ui мониторинга

-- Q13: Уникальные request_id по сообщениям
SELECT message, COUNT(DISTINCT request_id) AS u
FROM `log_workload_column`
WHERE message <> ''
GROUP BY message
ORDER BY u DESC
LIMIT 10

• а могут отработать на единичном запросе и начать опять падать только на потоке запросов bulk_upsert + select

Вот сейчас на кластере http://vla5-2566.search.yandex.net:8765/monitoring/cluster/nodes вы можете выполнить запрос приведенный выше и посмотреть упадут ли ноды ( у меня упали 3 минуты назад)

А в логах:

Feb 12 17:06:41 vla5-2573 kikimr_31003[651129]: Not freed 0x00007F0B8FB95F20 size: 32, location: yql/essentials/minikql/computation/mkql_computation_node_impl.h:972
Feb 12 17:06:41 vla5-2573 kikimr_31003[651129]: Not freed 0x00007F0BD6C8F610 size: 48, location: yql/essentials/minikql/computation/mkql_computation_node_holders.h:656
Feb 12 17:06:41 vla5-2573 kikimr_31003[651129]: Not freed 0x00007F0BD6C8F5F0 size: 32, location: yql/essentials/minikql/computation/mkql_computation_node_impl.h:972
Feb 12 17:06:41 vla5-2573 kikimr_31003[651129]: VERIFY failed (2025-02-12T17:06:13.995331+0300): Allocated: 171424, Freed: 170544, Peak: 11472
Feb 12 17:06:41 vla5-2573 kikimr_31003[651129]:   yql/essentials/minikql/mkql_mem_info.cpp:137
Feb 12 17:06:41 vla5-2573 kikimr_31003[651129]:   VerifyDebug(): requirement GetUsage() == 0 failed

also this error ocurred in #13992 ( log

VERIFY failed (2025-02-12T15:02:00.845046+0300): Allocated: 512792, Freed: 512664, Peak: 44312
  yql/essentials/minikql/mkql_mem_info.cpp:137
  VerifyDebug(): requirement GetUsage() == 0 failed
Not freed 0x00007F7398028350 size: 32, location: yql/essentials/minikql/computation/mkql_computation_node_holders.h:656
Not freed 0x00007F7398028330 size: 32, location: yql/essentials/minikql/computation/mkql_computation_node_impl.h:972
0. /-S/util/system/yassert.cpp:83: NPrivate::InternalPanicImpl(int, char const*, char const*, int, int, int, TBasicStringBuf<char, std::__y1::char_traits<char>>, char const*, unsigned long) @ 0x9B44D35
1. /-S/util/system/yassert.cpp:55: NPrivate::Panic(NPrivate::TStaticBuf const&, int, char const*, char const*, char const*, ...) @ 0x9B3F0EC
2. /-S/yql/essentials/minikql/mkql_mem_info.cpp:135: NKikimr::NMiniKQL::TMemoryUsageInfo::VerifyDebug() const @ 0xC346908
3. /-S/yql/essentials/minikql/mkql_mem_info.cpp:20: NKikimr::NMiniKQL::TMemoryUsageInfo::~TMemoryUsageInfo() @ 0xC3466DA
4. /-S/yql/essentials/minikql/mkql_mem_info.cpp:18: NKikimr::NMiniKQL::TMemoryUsageInfo::~TMemoryUsageInfo() @ 0xC34691D
5. /-S/util/generic/ptr.h:36: void CheckedDelete<TThrRefBase>(TThrRefBase*) @ 0xC3444D0
6. /-S/util/generic/ptr.h:57: void TDelete::Destroy<TThrRefBase>(TThrRefBase*) @ 0xC3444D0
7. /-S/util/generic/ptr.h:405: TRefCounted<TThrRefBase, TAtomicCounter, TDelete>::UnRef(long) @ 0xC3444D0
8. /-S/util/generic/ptr.h:410: TRefCounted<TThrRefBase, TAtomicCounter, TDelete>::UnRef() @ 0xC3444D0
9. /-S/util/generic/ptr.h:481: TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>::UnRef(NKikimr::NMiniKQL::TMemoryUsageInfo*) @ 0xC3444D0
10. /-S/util/generic/ptr.h:625: TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>::UnRef() @ 0xC3444D0
11. /-S/util/generic/ptr.h:523: TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>::~TIntrusivePtr() @ 0xC3444D0
12. /-S/contrib/libs/cxxsupp/libcxx/include/__utility/pair.h:63: std::__y1::pair<NKikimr::NMiniKQL::TMemoryUsageInfo* const, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>::~pair() @ 0xC3444D0
13. /-S/contrib/libs/cxxsupp/libcxx/include/__memory/construct_at.h:67: void std::__y1::__destroy_at[abi:fe190000]<std::__y1::pair<NKikimr::NMiniKQL::TMemoryUsageInfo* const, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>, 0>(std::__y1::pair<NKikimr::NMiniKQL::TMemoryUsageInfo* const, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>*) @ 0xC3444D0
14. /-S/contrib/libs/cxxsupp/libcxx/include/__memory/allocator_traits.h:339: void std::__y1::allocator_traits<std::__y1::allocator<std::__y1::__hash_node<std::__y1::__hash_value_type<NKikimr::NMiniKQL::TMemoryUsageInfo*, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>, void*>>>::destroy[abi:fe190000]<std::__y1::pair<NKikimr::NMiniKQL::TMemoryUsageInfo* const, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>, void, 0>(std::__y1::allocator<std::__y1::__hash_node<std::__y1::__hash_value_type<NKikimr::NMiniKQL::TMemoryUsageInfo*, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>, void*>>&, std::__y1::pair<NKikimr::NMiniKQL::TMemoryUsageInfo* const, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>*) @ 0xC3444D0
15. /-S/contrib/libs/cxxsupp/libcxx/include/__hash_table:1140: std::__y1::__hash_table<std::__y1::__hash_value_type<NKikimr::NMiniKQL::TMemoryUsageInfo*, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>, std::__y1::__unordered_map_hasher<NKikimr::NMiniKQL::TMemoryUsageInfo*, std::__y1::__hash_value_type<NKikimr::NMiniKQL::TMemoryUsageInfo*, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>, std::__y1::hash<NKikimr::NMiniKQL::TMemoryUsageInfo*>, std::__y1::equal_to<NKikimr::NMiniKQL::TMemoryUsageInfo*>, true>, std::__y1::__unordered_map_equal<NKikimr::NMiniKQL::TMemoryUsageInfo*, std::__y1::__hash_value_type<NKikimr::NMiniKQL::TMemoryUsageInfo*, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>, std::__y1::equal_to<NKikimr::NMiniKQL::TMemoryUsageInfo*>, std::__y1::hash<NKikimr::NMiniKQL::TMemoryUsageInfo*>, true>, std::__y1::allocator<std::__y1::__hash_value_type<NKikimr::NMiniKQL::TMemoryUsageInfo*, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>>>::__deallocate_node(std::__y1::__hash_node_base<std::__y1::__hash_node<std::__y1::__hash_value_type<NKikimr::NMiniKQL::TMemoryUsageInfo*, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>, void*>*>*) @ 0xC3444D0
16. /-S/contrib/libs/cxxsupp/libcxx/include/__hash_table:1316: std::__y1::__hash_table<std::__y1::__hash_value_type<NKikimr::NMiniKQL::TMemoryUsageInfo*, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>, std::__y1::__unordered_map_hasher<NKikimr::NMiniKQL::TMemoryUsageInfo*, std::__y1::__hash_value_type<NKikimr::NMiniKQL::TMemoryUsageInfo*, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>, std::__y1::hash<NKikimr::NMiniKQL::TMemoryUsageInfo*>, std::__y1::equal_to<NKikimr::NMiniKQL::TMemoryUsageInfo*>, true>, std::__y1::__unordered_map_equal<NKikimr::NMiniKQL::TMemoryUsageInfo*, std::__y1::__hash_value_type<NKikimr::NMiniKQL::TMemoryUsageInfo*, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>, std::__y1::equal_to<NKikimr::NMiniKQL::TMemoryUsageInfo*>, std::__y1::hash<NKikimr::NMiniKQL::TMemoryUsageInfo*>, true>, std::__y1::allocator<std::__y1::__hash_value_type<NKikimr::NMiniKQL::TMemoryUsageInfo*, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>>>::clear() @ 0xC3444D0
17. /-S/contrib/libs/cxxsupp/libcxx/include/unordered_map:1331: std::__y1::unordered_map<NKikimr::NMiniKQL::TMemoryUsageInfo*, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>, std::__y1::hash<NKikimr::NMiniKQL::TMemoryUsageInfo*>, std::__y1::equal_to<NKikimr::NMiniKQL::TMemoryUsageInfo*>, std::__y1::allocator<std::__y1::pair<NKikimr::NMiniKQL::TMemoryUsageInfo* const, TIntrusivePtr<NKikimr::NMiniKQL::TMemoryUsageInfo, TDefaultIntrusivePtrOps<NKikimr::NMiniKQL::TMemoryUsageInfo>>>>>::clear[abi:fe190000]() @ 0xC3444D0
18. /-S/yql/essentials/minikql/mkql_alloc.cpp:100: NKikimr::NMiniKQL::TAllocState::KillAllBoxed() @ 0xC3444D0
19. /-S/yql/essentials/minikql/mkql_alloc.h:212: NKikimr::NMiniKQL::TScopedAlloc::~TScopedAlloc() @ 0x1551E52A
20. /-S/contrib/libs/cxxsupp/libcxx/include/__memory/allocator.h:170: std::__y1::allocator<NKikimr::NMiniKQL::TScopedAlloc>::destroy[abi:fe190000](NKikimr::NMiniKQL::TScopedAlloc*) @ 0x1551E52A
21. /-S/contrib/libs/cxxsupp/libcxx/include/__memory/allocator_traits.h:334: void std::__y1::allocator_traits<std::__y1::allocator<NKikimr::NMiniKQL::TScopedAlloc>>::destroy[abi:fe190000]<NKikimr::NMiniKQL::TScopedAlloc, 0>(std::__y1::allocator<NKikimr::NMiniKQL::TScopedAlloc>&, NKikimr::NMiniKQL::TScopedAlloc*) @ 0x1551E52A
22. /-S/contrib/libs/cxxsupp/libcxx/include/__memory/shared_ptr.h:315: void std::__y1::__shared_ptr_emplace<NKikimr::NMiniKQL::TScopedAlloc, std::__y1::allocator<NKikimr::NMiniKQL::TScopedAlloc>>::__on_zero_shared_impl[abi:fe190000]<std::__y1::allocator<NKikimr::NMiniKQL::TScopedAlloc>, 0>() @ 0x1551E52A
23. /-S/contrib/libs/cxxsupp/libcxx/include/__memory/shared_ptr.h:318: std::__y1::__shared_ptr_emplace<NKikimr::NMiniKQL::TScopedAlloc, std::__y1::allocator<NKikimr::NMiniKQL::TScopedAlloc>>::__on_zero_shared() @ 0x1551E52A
24. /-S/contrib/libs/cxxsupp/libcxx/include/__memory/shared_ptr.h:171: std::__y1::__shared_count::__release_shared[abi:fe190000]() @ 0x19D66DD8
25. /-S/contrib/libs/cxxsupp/libcxx/include/__memory/shared_ptr.h:217: std::__y1::__shared_weak_count::__release_shared[abi:fe190000]() @ 0x19D66DD8
26. /-S/contrib/libs/cxxsupp/libcxx/include/__memory/shared_ptr.h:682: std::__y1::shared_ptr<NKikimr::NMiniKQL::TScopedAlloc>::~shared_ptr[abi:fe190000]() @ 0x19D66DD8
27. /-S/ydb/library/yql/dq/actors/compute/dq_compute_actor_impl.h:203: NYql::NDq::TDqComputeActorBase<NKikimr::NKqp::TKqpComputeActor, NYql::NDq::TComputeActorAsyncInputHelperSync>::~TDqComputeActorBase() @ 0x19D66DD8
28. /-S/ydb/core/kqp/compute_actor/kqp_pure_compute_actor.h:19: NKikimr::NKqp::TKqpComputeActor::~TKqpComputeActor() @ 0x19D5E52D
29. /-S/util/generic/ptr.h:36: void CheckedDelete<NActors::IActor>(NActors::IActor*) @ 0xA8A8E27
30. /-S/util/generic/ptr.h:57: void TDelete::Destroy<NActors::IActor>(NActors::IActor*) @ 0xA8A8E27
31. /-S/util/generic/ptr.h:360: THolder<NActors::IActor, TDelete>::DoDestroy() @ 0xA8A8E27
32. /-S/util/generic/ptr.h:290: THolder<NActors::IActor, TDelete>::~THolder() @ 0xA8A8E27
33. /-S/contrib/libs/cxxsupp/libcxx/include/__memory/allocator.h:170: std::__y1::allocator<THolder<NActors::IActor, TDelete>>::destroy[abi:fe190000](THolder<NActors::IActor, TDelete>*) @ 0xA8A8E27
34. /-S/contrib/libs/cxxsupp/libcxx/include/__memory/allocator_traits.h:334: void std::__y1::allocator_traits<std::__y1::allocator<THolder<NActors::IActor, TDelete>>>::destroy[abi:fe190000]<THolder<NActors::IActor, TDelete>, 0>(std::__y1::allocator<THolder<NActors::IActor, TDelete>>&, THolder<NActors::IActor, TDelete>*) @ 0xA8A8E27
35. /-S/contrib/libs/cxxsupp/libcxx/include/vector:939: std::__y1::vector<THolder<NActors::IActor, TDelete>, std::__y1::allocator<THolder<NActors::IActor, TDelete>>>::__base_destruct_at_end[abi:fe190000](THolder<NActors::IActor, TDelete>*) @ 0xA8A8E27
36. ```

[naspirato]

1. пересобрал бинарь shell ./ya make --build='release' ydb/apps/ydbd/
2. налил на кластер (http://vla5-2566.search.yandex.net:8765/monitoring/cluster/nodes) где уже были падения и была база от предыдущей сборки через shell ./ydbd_slice update $LONG all --binary=/home/kirrysin/ydbd_main_release
3. зашел в ui http://vla5-2566.search.yandex.net:8765/monitoring/tenant?tenantPage=query&queryTab=newQuery&database=%2FRoot%2Fdb1
4. выполнил SCAN запрос с timeout 120 sec

-- Q13: Уникальные request_id по сообщениям
SELECT message, COUNT(DISTINCT request_id) AS u
FROM `log_workload_column`
WHERE message <> ''
GROUP BY message
ORDER BY u DESC
LIMIT 10

5. получил
   в UI

error
detected disconnected node

В логах - пусто
в minidumn - были, улетели в корки, вот они

• https://coredumps.yandex-team.ru/core?core_hash=1148933822079060497&itype=kikimr
• https://coredumps.yandex-team.ru/core?core_hash=37107065514069106&itype=kikimr
• https://coredumps.yandex-team.ru/core?core_hash=845458452182647055&itype=kikimr
• https://coredumps.yandex-team.ru/core?core_hash=940983118769830659&itype=kikimr
• https://coredumps.yandex-team.ru/core?core_hash=610088608726562642&itype=kikimr

[lll-phill-lll]

This should be the fix:
https://github.com/ydb-platform/ydb/pull/14593/files

Docs with detailed explanations will be provided later (hopefully today).

[zverevgeny]

Заметки со встречи. Пока оставляем в проекте ColumnShard. @zverevgeny подумает и предложит способ трекать проблемы Аналитики в разных проектах

[naspirato]

#14593