[A2/A3] Tile::isKAligned_ may be read before initialization

[Little-oil]

Summary

Tile::isKAligned_ is not initialized by its declaration or by any Tile constructor, but several A2/A3 paths read it through GetKAligned() without first guaranteeing that SetKAligned() has been called.

Under normal C++ object-initialization rules, the member therefore has an indeterminate value. Reading it may result in undefined or build-dependent behavior.

The issue is present on current main and also in the PTO-ISA commit currently used by PyPTO CI (f51c92f).

Relevant code

The constructors explicitly initialize data_ in auto mode to prevent it from becoming undef after SROA, but do not initialize isKAligned_:

AICORE Tile()
{
#if defined(__PTO_AUTO__) && !defined(__CPU_SIM)
    // Prevent data_ from becoming undef after SROA.
    data_ = __cce_tinit(data_);
#endif
}

The getter directly returns the uninitialized member:

PTO_INTERNAL bool GetKAligned() const
{
    return isKAligned_;
}

PTO_INTERNAL void SetKAligned(bool isKAligned)
{
    isKAligned_ = isKAligned;
}

private:
    bool isKAligned_;

The same omission exists in the constructors for dynamic valid rows and columns.

Current source:

pto-isa/include/pto/common/pto_tile.hpp

Lines 1458 to 1703 in 60081f3

	{
	#if defined(__PTO_AUTO__) && !defined(__CPU_SIM)
	// we need to dummy-initialize the data_ member,
	// otherwise in auto mode this will remain uninitialized
	// and end up being an undef value after SROA pass
	data_ = __cce_tinit(data_);
	#endif
	};
	// constructor for both dimensions are runtime variables
	template <int RowMask = ValidRow, int ColMask = ValidCol>
	AICORE Tile(
	std::enable_if_t<RowMask == DYNAMIC && ColMask == DYNAMIC, unsigned> VR,
	std::enable_if_t<RowMask == DYNAMIC && ColMask == DYNAMIC, unsigned> VC)
	{
	#if defined(__PTO_AUTO__) && !defined(__CPU_SIM)
	data_ = __cce_tinit(data_);
	#endif
	RowMaskInternal = VR;
	ColMaskInternal = VC;
	}
	// constructor for row dimension is runtime variables
	template <int RowMask = ValidRow, int ColMask = ValidCol>
	AICORE Tile(std::enable_if_t<(RowMask == DYNAMIC) && (ColMask > 0), unsigned> VR)
	{
	#ifdef __PTO_AUTO__
	data_ = __cce_tinit(data_);
	#endif
	RowMaskInternal = VR;
	}
	// constructor for col dimension is runtime variables
	template <int RowMask = ValidRow, int ColMask = ValidCol>
	AICORE Tile(std::enable_if_t<(RowMask > 0) && (ColMask == DYNAMIC), unsigned> VC)
	{
	#ifdef __PTO_AUTO__
	data_ = __cce_tinit(data_);
	#endif
	ColMaskInternal = VC;
	}
	#ifdef __PTO_AUTO__
	Tile& operator=(const Tile&) = delete;
	Tile& operator=(Tile&&) = delete;
	#endif
	static constexpr bool isBoxedLayout = (SFractal != SLayout::NoneBox);
	static constexpr bool isInnerRowMajor = (SFractal == SLayout::RowMajor);
	static constexpr bool isInnerColMajor = (SFractal == SLayout::ColMajor);
	static constexpr int InnerRows = getInnerRow();
	static constexpr int InnerCols = getInnerCol();
	static constexpr int InnerNumel = InnerRows * InnerCols;
	static_assert(InnerRows != 0 && InnerCols != 0, "rows or cols of fractal size is 0.");
	static_assert(
	(Loc == TileType::Vec) || (SFractalSize_ == TileConfig::fractalMxSize) || (Rows_ == 1) ||
	(Rows % InnerRows == 0),
	"Layout rows must be divisible by inner box rows");
	static_assert(Cols % InnerCols == 0, "Layout cols must be divisible by inner box cols");
	static_assert(
	(BFractal_ == BLayout::RowMajor && SFractal_ == SLayout::NoneBox &&
	Cols * sizeof(DType) % TileConfig::alignedSize == 0) ||
	(BFractal_ == BLayout::ColMajor && SFractal_ == SLayout::NoneBox &&
	Rows * sizeof(DType) % TileConfig::alignedSize == 0) ||
	(SFractal_ != SLayout::NoneBox) &&
	(((Loc == TileType::Vec) || (SFractalSize_ == TileConfig::fractalMxSize) || (Rows_ == 1) ||
	(Rows % InnerRows == 0)) &&
	Cols % InnerCols == 0),
	"BFractal_ is RowMajor and SFractal_ is NoneBox: Rows must be 32 bytes align, \
	BFractal_ is ColMajor and SFractal_ is NoneBox: Cols must be 32 bytes align, \
	SFractal_ in not NoneBox: Rows/Cols must be integer multiple of InnerRows/InnerCols.");
	static_assert(
	SFractalSize_ == TileConfig::fractalABSize || SFractalSize_ == TileConfig::fractalCSize ||
	SFractalSize_ == TileConfig::fractalMxSize,
	"SFractalSize_ illegal");
	#if defined(__CPU_SIM) || defined(__COSTMODEL)
	// CPU Sim: data_ is a pointer that TASSIGN can redirect to shared NPU memory
	using TileDType = Tile::DType*;
	#else
	#ifdef __PTO_AUTO__
	#if defined(PTO_NPU_ARCH_A2A3)
	using TileDType = typename MemoryQualifier<Loc, DType>::type tile_size(Rows* Cols);
	#else
	using TileDType = std::conditional_t<
	Loc == TileType::Bias, typename MemoryQualifier<Loc, DType>::type, // special handling for Bias Tile
	typename MemoryQualifier<Loc, DType>::type tile_size(Rows* Cols)>;
	#endif
	#else
	using TileDType = typename MemoryQualifier<Loc, DType>::type;
	#endif
	#endif
	#if (defined(__CPU_SIM) && defined(__PTO_AUTO__)) || defined(__COSTMODEL)
	TileDType& data()
	{
	if (!data_) {
	internalBuffer.resize(Rows * Cols / (IsTwinType<DType>() ? 2 : 1));
	data_ = internalBuffer.data();
	}
	return data_;
	}
	const TileDType& data() const
	{
	if (!data_) {
	internalBuffer.resize(Rows * Cols);
	data_ = internalBuffer.data();
	}
	return data_;
	}
	#else
	AICORE TileDType& data() { return data_; }
	AICORE const TileDType& data() const { return data_; }
	#endif
	#ifdef __COSTMODEL
	float cycle;
	AICORE void SetCycle(const float cycle_) { cycle = cycle_; }
	AICORE void SetLastCycle(const float cycle_) { cycle = cycle_; }
	AICORE float GetCycle() { return cycle; }
	#endif
	unsigned RowMaskInternal;
	unsigned ColMaskInternal;
	template <int RowMask = ValidRow>
	AICORE static constexpr std::enable_if_t<(RowMask > 0), unsigned> GetValidRow()
	{
	return RowMask;
	}
	template <int RowMask = ValidRow>
	AICORE std::enable_if_t<RowMask == DYNAMIC, unsigned> GetValidRow() const
	{
	return RowMaskInternal;
	}
	template <int ColMask = ValidCol>
	AICORE static constexpr std::enable_if_t<(ColMask > 0), unsigned> GetValidCol()
	{
	return ColMask;
	}
	template <int ColMask = ValidCol>
	AICORE std::enable_if_t<ColMask == DYNAMIC, unsigned> GetValidCol() const
	{
	return ColMaskInternal;
	}
	// Call this function need PIPE_S wait
	PTO_INTERNAL void SetValidRow(unsigned rowMask)
	{
	static_assert(ValidRow == DYNAMIC, "Only Dynamic Valid Row Support Set Value.");
	PTO_ASSERT(rowMask <= Rows, "rowMask must less than Rows.");
	RowMaskInternal = rowMask;
	}
	// Call this function need PIPE_S wait
	PTO_INTERNAL void SetValidCol(unsigned colMask)
	{
	static_assert(ValidCol == DYNAMIC, "Only Dynamic Valid Col Support Set Value.");
	PTO_ASSERT(colMask <= Cols, "colMask must less than Cols.");
	ColMaskInternal = colMask;
	}
	// Call this function need PIPE_S wait
	PTO_INTERNAL void SetValidShape(unsigned rowMask, unsigned colMask)
	{
	static_assert(ValidCol == DYNAMIC && ValidRow == DYNAMIC, "Only Dynamic Valid Shape Support Set Value.");
	PTO_ASSERT(rowMask <= Rows && colMask <= Cols, "rowMask and colMask must not exceed Rows and Cols.");
	RowMaskInternal = rowMask;
	ColMaskInternal = colMask;
	}
	template <typename T, typename AddrType>
	friend AICORE void TASSIGN_IMPL(T& tile, AddrType addr);
	PTO_INTERNAL bool GetKAligned() const { return isKAligned_; }
	PTO_INTERNAL void SetKAligned(bool isKAligned) { isKAligned_ = isKAligned; }
	#if defined(__DAV_CUBE__)
	/*
	TF32 precision implementation varies across different chips:
	- a2/a3 : e8m11(1 sign bits, 8 exponent bits, 11 mantissa bits)
	- a5 : e8m10(1 sign bits, 8 exponent bits, 10 mantissa bits)
	*/
	PTO_INTERNAL void SetMadTF32Mode(RoundMode tf32TransMode = RoundMode::CAST_ROUND)
	{
	PTO_ASSERT(
	tf32TransMode == RoundMode::CAST_ROUND || tf32TransMode == RoundMode::CAST_RINT,
	"Unsupported RoundMode for TF32.");
	set_ctrl(sbitset1(get_ctrl(), MAD_MODE_BIT));
	if (tf32TransMode == RoundMode::CAST_ROUND) {
	set_ctrl(sbitset1(get_ctrl(), MAD_ROUND_MODE_BIT));
	} else if (tf32TransMode == RoundMode::CAST_RINT) {
	set_ctrl(sbitset0(get_ctrl(), MAD_ROUND_MODE_BIT));
	}
	}
	PTO_INTERNAL void ResetMadMode() { set_ctrl(sbitset0(get_ctrl(), MAD_MODE_BIT)); }
	#endif
	#if defined(__CPU_SIM)
	static constexpr size_t GetSizeInUnits()
	{
	// One unit is sizeof(DType)
	if constexpr (IsTwinType<DType>()) {
	return Numel / 2;
	} else {
	return Numel;
	}
	}
	static constexpr size_t GetSizeInBytes() { return GetSizeInUnits() * sizeof(DType); }
	DType GetElement(int64_t r, int64_t c)
	{
	return GetProperDataPart(data(), GetTileElementOffset<std::remove_reference_t<decltype(*this)>>(r, c));
	}
	void SetElement(int64_t r, int64_t c, const DType& val)
	{
	const auto offset = GetTileElementOffset<std::remove_reference_t<decltype(*this)>>(r, c);
	SetProperDataPart(data(), offset, val);
	}
	void AddToElement(int64_t r, int64_t c, const DType& summand)
	{
	const auto offset = GetTileElementOffset<std::remove_reference_t<decltype(*this)>>(r, c);
	std::lock_guard<std::mutex> lock(cpu::AtomicAddMutex());
	if constexpr (IsTwinType<DType>()) {
	const auto val = GetProperDataPart(data(), offset);
	SetProperDataPart(data(), offset, val + summand);
	} else {
	data()[offset] += summand;
	}
	}
	#endif
	private:
	AICORE void assignData(TileDType data) { data_ = data; }
	bool isKAligned_; // K alignment flag for A3.

Read paths

On A2/A3, the FP32 matmul/GEMV path reads the flag for both input tiles:

if constexpr (
    std::is_same<typename TileLeft::DType, float>::value &&
    std::is_same<typename TileRight::DType, float>::value) {
    bool cond = aMatrix.GetKAligned() || bMatrix.GetKAligned();
    return cond;
}

The returned value is passed to the mad intrinsic as kDirectionAlign.

Source:

pto-isa/include/pto/npu/a2a3/TMatmul.hpp

Lines 20 to 34 in 60081f3

	template <typename TileLeft, typename TileRight>
	PTO_INTERNAL bool GetKDirectionAlign(TileLeft& aMatrix, TileRight& bMatrix)
	{
	// only for f322f32
	// #ifndef __PTO_AUTO__
	if constexpr (
	std::is_same<typename TileLeft::DType, float>::value && std::is_same<typename TileRight::DType, float>::value) {
	bool cond = aMatrix.GetKAligned() || bMatrix.GetKAligned();
	if (cond) {
	return true;
	}
	return false;
	}
	// #endif
	return false;

Other compact TMOV/TEXTRACT and communication paths also read or propagate GetKAligned(), so the problem is not limited to one GEMV testcase.

Potential impact

For a default-constructed tile that has never received an explicit SetKAligned() call, the effective K-alignment mode is not defined.

Possible outcomes include:

• different kDirectionAlign values across builds or executions;
• incorrect selection of an A2/A3 FP32 MAD mode;
• numerical errors or device-side stalls;
• behavior that remains consistently correct in one environment but changes with stack/register allocation or compiler optimization.

The exact hardware consequence of an unexpected true value still needs confirmation.

Suspected downstream symptom

This was found while investigating the remaining intermittent failure in PyPTO PR #2396:

hw-native-sys/pypto#2396

The PR now emits the matching final store phase, but system-tests-direct still occasionally fails in:

TestGemvAcc::test_tile_gemv_acc_partial_final_phases
tile_gemv_acc_1x256x64_fp32_chunks2_partial_final

The observed failure is:

ACL_ERROR_RT_AICPU_EXCEPTION: 507018
sched_error_code=100
sub_class=S1:running-stalled
completed=0/1
running=1

Example CI job:

https://github.com/hw-native-sys/pypto/actions/runs/32112775934/job/95636999561

The failing testcase uses FP32, so it executes the GetKDirectionAlign() path that reads isKAligned_. However, this connection is currently a hypothesis, not a confirmed causal result.

Local experiment

I compared the same repeated workload with:

bool isKAligned_;

and:

bool isKAligned_ = false;

Both variants passed 64/64 executions on the current server.

Therefore:

• the intermittent failure has not been reproduced locally;
• the initialization change has not yet demonstrated a behavioral difference;
• an equivalent A/B experiment is being continued in the CI environment.

The local pass does not eliminate the source-level defect because an uninitialized scalar is not guaranteed to produce a different value in every environment.

Expected behavior

A newly constructed tile should default to the ordinary, non-special K-alignment mode unless a caller explicitly opts in through SetKAligned(true):

bool isKAligned_ = false;

An in-class initializer covers all current and future constructors.

Suggested validation

• Verify GetKAligned() returns false for default-constructed static and dynamic-shape tiles.
• Verify SetKAligned(true) still returns true.
• Cover the CPU simulator and A2/A3 auto-mode builds.
• Compile an FP32 TMATMUL/TGEMV kernel and ensure no undef value can reach the kDirectionAlign argument.
• Run repeated A2/A3 FP32 GEMV/matmul tests before and after initialization.
• Run the PyPTO batched direct-device workload to check whether the intermittent scheduler stall disappears.

[Crystal-wzy]

Fixed in https://gitcode.com/cann/pto-isa/pull/1498.